Ceph 分布式存储 对象存储管理

Ceph 分布式存储 对象存储管理

对象存储是一个基于对象的存储服务, 提供海量、 安全、 高可靠、 低成本的数据存储能力

RADOS 网关介绍

RADOS GateWay,也称对象网关 (RGW),为客户端提供标准对象存储 API 来访问 Ceph 集群

守护进程 radosgw 在 librados 库的基础上构建

radosgw 是Ceph 存储的客户端,用于访问对象存储中对象

RADOS 网关架构

区域(zone),每个区域会关联一个或多个 RADOS 网关,这些网关关联Ceph 存储

区域组(zone group ),区域组是由一个或多个区域的集合。存储在区域组中某个区域中的数据会被复制到该区域组中的所有其他区域

域 (realm) ,代表所有对象和存储桶的全局命名空间

RADOS 网关部署

bash 复制代码
# 创建 realm
[root@ceph1 ~]# radosgw-admin realm create --rgw-realm=webapp --default
[root@ceph1 ~]# radosgw-admin realm list
{
"default_info": "527e3bea-d51d-462e-92d7-79aaf422a89c",
"realms": [
"webapp"
]
}
# 创建master zone group
[root@ceph1 ~]# radosgw-admin zonegroup create --rgw-realm=webapp --rgw-zonegroup=video --master --default
[root@ceph1 ~]# radosgw-admin zonegroup list
{
[
"default_info": "0fdaf584-33ee-4be7-978c-fb2771181aee",
"zonegroups":
"video"
 ]
}
# 创建 zone,并将其设置为 master
[root@ceph1 ~]# radosgw-admin zone create --rgw-realm=webapp --rgw-zonegroup=video --rgw-zone=storage1 --master --default
[root@ceph1 ~]# radosgw-admin zone list
{
"default_info": "76366a48-2ab8-4830-9cf7-f9b2fe432c5a",
"zones": [
"storage1"
]
}
# 提交配置
[root@ceph1 ~]# radosgw-admin period update --rgw-realm=webapp --commit
# 查看配置
[root@ceph1 ~]# radosgw-admin zone get --rgw-zone=storage1

RADOS 网关部署-命令行

bash 复制代码
# 创建 rgw,并和已创建的 realm webapp 进行关联,数量为 3
[root@ceph1 ~]# ceph orch apply rgw webapp --placement="3 ceph1.laogao.cloud 
ceph2.laogao.cloud ceph3.laogao.cloud" --realm=webapp --zone=storage1 --port=8080
# 查看服务
[root@ceph1 ~]# ceph orch ls rgw
NAME        
PORTS   
rgw.webapp  ?:8080      
RUNNING  REFRESHED  AGE  PLACEMENT
3/3  40s ago    45s  
ceph1.laogao.cloud;ceph2.laogao.cloud;ceph3.laogao.cloud;count:3
# 查看进程
[root@ceph1 ~]# ceph orch ps --daemon-type rgw| awk '{print $1,$4}'
NAME STATUS
rgw.webapp.ceph1.luxaau running
rgw.webapp.ceph2.hrdzlb running
rgw.webapp.ceph3.pyfond running
验证对象存储是否可以访问
[root@ceph1 ~]# curl http://ceph1.laogao.cloud:8080
<?xml version="1.0" encoding="UTF-8"?><ListAllMyBucketsResult 
xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Owner><ID>anonymous</ID>
<DisplayName></DisplayName></Owner><Buckets></Buckets></ListAllMyBucketsResult>
[root@ceph1 ~]# curl http://ceph2.laogao.cloud:8080
<?xml version="1.0" encoding="UTF-8"?><ListAllMyBucketsResult 
xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Owner><ID>anonymous</ID>
<DisplayName></DisplayName></Owner><Buckets></Buckets></ListAllMyBucketsResult>
[root@ceph1 ~]# curl http://ceph3.laogao.cloud:8080
<?xml version="1.0" encoding="UTF-8"?><ListAllMyBucketsResult 
xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Owner><ID>anonymous</ID>
<DisplayName></DisplayName></Owner><Buckets></Buckets></ListAllMyBucketsResult>

RADOS 网关删除

删除 RADOS 网关服务,将删除 RADOS 网关对应所有进程

bash 复制代码
[root@ceph1 ~]# ceph orch rm rgw.webapp

删除RADOS 网关,并不会删除池中数据

删除对象存储域

bash 复制代码
# 删除 zone
[root@ceph1 ~]# radosgw-admin zone delete --rgw-zone=storage1
# 删除 zonegroup
[root@ceph1 ~]# radosgw-admin zonegroup delete --rgw-zonegroup=video
# 删除 realm
[root@ceph1 ~]# radosgw-admin realm delete --rgw-realm=webapp

管理对象网关用户

创建用户

使用 radosgw-admin user create 命令来创建 RADOS 网关用户

bash 复制代码
[root@ceph1 ~]# radosgw-admin user create --uid="operator" --display-name="S3 
Operator" --email="operator@example.com" --access-key="12345" --secret
key="67890"
{
"user_id": "operator",
"display_name": "S3 Operator",
"email": "operator@example.com",
"suspended": 0,
"max_buckets": 1000,
"subusers": [],
"keys": [
{
"user": "operator",
"access_key": "12345",
"secret_key": "67890"
}
],
......
[root@ceph1 ~]# radosgw-admin user list
[
"operator",
"dashboard"
]
[root@ceph1 ~]# radosgw-admin user info --uid=operator
{
"user_id": "operator",
"display_name": "S3 Operator",
"email": "operator@example.com",
 "suspended": 0,
    "max_buckets": 1000,
    "subusers": [],
    "keys": [
        {
            "user": "operator",
            "access_key": "12345",
            "secret_key": "67890"
        }
    ],

如果未指定访问密钥和机密密钥,则自动生成,并显示在输出中

重新生成密钥

仅重新生成现有用户的机密密钥

bash 复制代码
[root@ceph1 ~]# radosgw-admin key create --uid=s3user --access
key="ZQI72JZZDTA8BRCQOLGK" --gen-secret
{
    "user_id": "s3user",
    "display_name": "Amazon S3 API user",
    "email": "",
    "suspended": 0,
    "max_buckets": 1000,
    "subusers": [],
    "keys": [
        {
            "user": "s3user",
            "access_key": "ZQI72JZZDTA8BRCQOLGK",
            "secret_key": "TJCcwo8rspE4xYyLAZHz4vapWA42YGB8hhrlPUnl"
        }

添加用户访问密钥

bash 复制代码
[root@ceph1 ~]# radosgw-admin key create --uid=s3user --gen-access-key
{
    "user_id": "s3user",
    "display_name": "Amazon S3 API user",
    "email": "",
    "suspended": 0,
    "max_buckets": 1000,
    "subusers": [],
    "keys": [
        {
            "user": "s3user",
            "access_key": "6WBR1FL8CBFX26KVRPPA",
            "secret_key": "UsytqYCeSudSqHKWwVVYZGFDdcS24lPpkXBrD42G"
        },
        {
            "user": "s3user",
            "access_key": "ZQI72JZZDTA8BRCQOLGK",
            "secret_key": "TJCcwo8rspE4xYyLAZHz4vapWA42YGB8hhrlPUnl"
        }
    ],
......

删除用户密钥

bash 复制代码
[root@ceph1 ~]# radosgw-admin key rm --uid=s3user --access
key=6WBR1FL8CBFX26KVRPPA
{
    "user_id": "s3user",
    "display_name": "Amazon S3 API user",
    "email": "",
    "suspended": 0,
    "max_buckets": 1000,
    "subusers": [],
    "keys": [
        {
            "user": "s3user",
            "access_key": "ZQI72JZZDTA8BRCQOLGK",
            "secret_key": "TJCcwo8rspE4xYyLAZHz4vapWA42YGB8hhrlPUnl"
        }
    ],
......

临时禁用对象网关用户

使用 radosgw-admin user suspend 命令可临时禁用 RADOS 网关用户,用户的子用户也会暂停,无法

与 RADOS 网关服务交互

bash 复制代码
root@ceph1 ~]# radosgw-admin user suspend --uid=s3user
{
"user_id": "s3user",
"display_name": "Amazon S3 API user",
"email": "",
"suspended": 1,     <----看着变成了1                 
....

临时启用对象网关用户

使用 radosgw-admin user enable 命令可启用 RADOS 网关用户

bash 复制代码
root@ceph1 ~]# radosgw-admin user enable --uid=s3user
{
"user_id": "s3user",
"display_name": "Amazon S3 API user",
"email": "",
"suspended": 0,
....

修改用户信息

bash 复制代码
[root@ceph1 ~]# radosgw-admin user modify --uid=s3user --display-name=huitailang
[root@ceph1 ~]# radosgw-admin user modify --uid=s3user --access=full

删除用户

bash 复制代码
[root@ceph1 ~]# radosgw-admin user list
[
"operator",
"dashboard",
"s3user"
]
[root@ceph1 ~]# radosgw-admin user rm --uid=s3user --purge-data
[root@ceph1 ~]# radosgw-admin user list
[
"operator",
"dashboard"
]

使用 Amazon S3 API 访问对象存储

安装 Amazon S3 API 客户端

bash 复制代码
# 准备 aliyun 源 pip 仓库
[root@client ~]# mkdir .pip
[root@client ~]# cat > .pip/pip.conf << 'EOF'
[global]
index-url = http://mirrors.aliyun.com/pypi/simple/
[install]
trusted-host=mirrors.aliyun.com
EOF
# 安装软件包
[root@client ~]# pip3 install awscli

使用 Amazon S3 API 客户端

配置 AWS CLI 凭据
bash 复制代码
[root@client ~]# aws configure
AWS Access Key ID [None]: 12345 
AWS Secret Access Key [None]: 67890
Default region name [None]: 
Default output format [None]: 

配置信息保存在~/.aws目录。

bash 复制代码
[root@client ~]# ls .aws
config  credentials
[root@client ~]# cat .aws/config 
[default]
[root@client ~]# cat .aws/credentials 
[default]
aws_access_key_id = 12345
aws_secret_access_key = 67890
创建存储桶

环境准备:

client:

bash 复制代码
[root@client ~]# scp 192.168.108.11:/etc/hosts /etc/hosts

可以通过选项 --profile 指定凭据,--endpoint 指定服务器位置

bash 复制代码
[root@client ~]# aws --endpoint=http://ceph1.laogao.cloud:8080 s3 mb s3://webapp
make_bucket: webapp
查看存储桶清单
bash 复制代码
[root@client ~]# aws --endpoint=http://ceph1.laogao.cloud:8080 s3 ls
2025-08-25 17:14:05 webapp

或者

bash 复制代码
[root@ceph1 ~]# radosgw-admin buckets list
[
"webapp"
]
上传对象到存储桶
bash 复制代码
[root@client ~]# echo Hello World > Welcome-pub.html
[root@client ~]# echo Hello laogao > Welcome-pri.html
[root@client ~]# aws --endpoint=http://ceph1.laogao.cloud:8080 s3 cp Welcome
pub.html s3://webapp/ --acl=public-read-write
upload: ./Welcome-pub.html to s3://webapp/Welcome-pub.html
[root@client ~]# aws s3 cp Welcome-pri.html s3://webapp --
endpoint=http://ceph1.laogao.cloud:8080
upload: ./Welcome-pri.html to s3://webapp/Welcome-pri.html
查看存储桶中对象
bash 复制代码
[root@client ~]# aws s3 ls s3://webapp --endpoint=http://ceph1.laogao.cloud:8080
2025-08-25 17:23:35    13 Welcome-pri.html
2025-08-25 17:22:18    12 Welcome-pub.html
下载存储桶中对象
bash 复制代码
# 下载单个
[root@client ~]# aws s3 cp s3://webapp/Welcome-pri.html /tmp --
endpoint=http://ceph1.laogao.cloud:8080
download: s3://webapp/Welcome-pri.html to ../tmp/Welcome-pri.html
[root@client ~]# ls /tmp/Welcome-pri.html
/tmp/Welcome-pri.html
# 使用选项 --recursive 递归下载bucket中所有对象
[root@client ~]# aws s3 cp s3://webapp /tmp --recursive --
endpoint=http://ceph1.laogao.cloud:8080
download: s3://webapp/Welcome-pri.html to ../tmp/Welcome-pri.html
download: s3://webapp/Welcome-pub.html to ../tmp/Welcome-pub.html
# 使用wget或者curl下载
[root@client ~]# wget http://ceph1.laogao.cloud:8080/webapp/Welcome-pub.html   # Welcome-pub.html可以正常下载
[root@client ~]# wget http://ceph1.laogao.cloud:8080/webapp/Welcome-pri.html   # Welcome-pri.html不可以下载,因为权限不允许--2025-08-25 17:36:http://ceph1.laogao.cloud:8080/webapp/Welcome-pri.html
Resolving ceph1.laogao.cloud (ceph1.laogao.cloud)... 192.168.108.11
Connecting to ceph1.laogao.cloud (ceph1.laogao.cloud)|192.168.108.11|:8080... 
connected.
HTTP request sent, awaiting response... 403 Forbidden
2025-08-25 17:36:59 ERROR 403: Forbidden.
删除存储桶中对象
bash 复制代码
[root@client ~]# aws s3 rm s3://webapp/Welcome-pri.html --
endpoint=http://ceph1.laogao.cloud:8080
delete: s3://webapp/Welcome-pri.html
[root@client ~]# aws s3 ls s3://webapp --endpoint=http://ceph1.laogao.cloud:8080
2025-08-25 17:22:18         12 Welcome-pub.html
删除存储桶
bash 复制代码
# 非空的存储桶不能删除
[root@client ~]# aws s3 rb s3://webapp --endpoint=http://ceph1.laogao.cloud:8080
remove_bucket failed: s3://webapp argument of type 'NoneType' is not iterable
# 清空存储桶,再次删除
[root@client ~]# aws s3 rm s3://webapp --recursive --include "Welcome-*" --
endpoint=http://ceph1.laogao.cloud:8080
delete: s3://webapp/Welcome-pub.html
[root@client ~]# aws s3 rb s3://webapp --endpoint=http://ceph1.laogao.cloud:8080
remove_bucket: webapp

用户配额管理

设置配额可限制用户或存储桶可消耗的存储量

user 级别
bash 复制代码
[root@ceph1 ~]# radosgw-admin quota enable --quota-scope=user --uid=operator
[root@ceph1 ~]# radosgw-admin quota set --quota-scope=user --uid=operator --max
objects=1024
# 查看用户配额
[root@ceph1 ~]# radosgw-admin user info --uid=operator
{
    "user_id": "operator",
    "display_name": "S3 Operator",
    "email": "operator@example.com",
    "suspended": 0,
    "max_buckets": 1000,
    "subusers": [],
    "keys": [
        {
            "user": "operator",
            "access_key": "12345",
            "secret_key": "67890"
        }
    ],
    "swift_keys": [],
    "caps": [],
    "op_mask": "read, write, delete",
    "default_placement": "",
    "default_storage_class": "",
    "placement_tags": [],
    "bucket_quota": {
        "enabled": false,
        "check_on_raw": false,
        "max_size": -1,
        "max_size_kb": 0,
        "max_objects": -1
    },
    "user_quota": {
        "enabled": true,
        "check_on_raw": false,
        "max_size": -1,
        "max_size_kb": 0,
        "max_objects": 1024
    },
    "temp_url_keys": [],
    "type": "rgw",
    "mfa_ids": []
}
# 查看空间使用情况
[root@ceph1 ~]# radosgw-admin user stats --uid=operator
{
}
"stats": {
"size": 0,
"size_actual": 0,
"size_kb": 0,
"size_kb_actual": 0,
"num_objects": 0
},
"last_stats_sync": "0.000000",
"last_stats_update": "2025-08-25T09:44:36.609841Z"
# 测试
[root@client ~]# aws --endpoint=http://ceph1.laogao.cloud:8080 s3 mb s3://webapp
make_bucket: webapp
[root@client ~]# aws --endpoint=http://ceph1.laogao.cloud:8080 s3 ls s3://webapp 
#先看下webapp桶有多少文件,现在没有
# 传3各文件试下
[root@client ~]# touch file1 file2 file3 file4
[root@client ~]# aws --endpoint=http://ceph1.laogao.cloud:8080 s3 cp file1 
s3://webapp
upload: ./file1 to s3://webapp/file1
[root@client ~]# aws --endpoint=http://ceph1.laogao.cloud:8080 s3 cp file2 
s3://webapp
upload: ./file2 to s3://webapp/file2
[root@client ~]# aws --endpoint=http://ceph1.laogao.cloud:8080 s3 cp file3 
s3://webapp
upload: ./file3 to s3://webapp/file3
[root@client ~]# aws --endpoint=http://ceph1.laogao.cloud:8080 s3 ls s3://webapp
2026-06-02 09:44:58       0 file1   
2026-06-02 09:45:05       0 file2   
2026-06-02 09:45:10       0 file3   
[root@ceph1 ~]# radosgw-admin user stats --uid=operator
{
"stats": {
"size": 0,
"size_actual": 0,
"size_kb": 0,
"size_kb_actual": 0,
"num_objects": 3                          
#已经达到最大3个
},
"last_stats_sync": "2026-06-02T01:17:04.891186Z",
"last_stats_update": "2026-06-02T01:47:04.831294Z"
}
# 第四个文件无法上传
[root@client ~]# aws --endpoint=http://ceph1.laogao.cloud:8080 s3 cp file4 
s3://webapp
upload failed: ./file4 to s3://webapp/file4 argument of type 'NoneType' is not 
iterable

禁用用户配额

示例:禁用operator用户配额

bash 复制代码
[root@ceph1 ~]# radosgw-admin quota set --quota-scope=user --uid=operator --max
objects=-1
[root@ceph1 ~]# radosgw-admin quota disable --quota-scope=user --uid=operator
[root@ceph1 ~]# radosgw-admin user info --uid=operator
{
    "user_id": "operator",
    "display_name": "S3 Operator",
    "email": "operator@example.com",
    "suspended": 0,
    "max_buckets": 1000,
    "subusers": [],
    "keys": [
        {
            "user": "operator",
            "access_key": "12345",
            "secret_key": "67890"
        }
    ],
    "swift_keys": [],
    "caps": [],
    "op_mask": "read, write, delete",
    "default_placement": "",
    "default_storage_class": "",
    "placement_tags": [],
    "bucket_quota": {
        "enabled": false,
        "check_on_raw": false,
        "max_size": -1,
        "max_size_kb": 0,
        "max_objects": -1
    },
    "user_quota": {
        "enabled": false,
        "check_on_raw": false,
        "max_size": -1,
        "max_size_kb": 0,
        "max_objects": -1
    },
    "temp_url_keys": [],
    "type": "rgw",
    "mfa_ids": []
}
# 再次上传第四个文件测试
[root@client ~]# aws --endpoint=http://ceph1.laogao.cloud:8080 s3 cp file4 
s3://webapp
upload: ./file4 to s3://webapp/file4
bucket 级别

:webapp 存储桶被设置为最多 1024 个对象

bash 复制代码
[root@client ~]# aws --endpoint=http://ceph1.laogao.cloud:8080 s3 mb s3://webapp
make_bucket: webapp
[root@ceph1 ~]# radosgw-admin quota enable --quota-scope=bucket --bucket=webapp
[root@ceph1 ~]# radosgw-admin quota set --quota-scope=bucket --bucket=webapp --
max-objects=1024
# 查看 bucket 配额
[root@ceph1 ~]# radosgw-admin bucket stats --bucket=webapp
{
"bucket": "webapp",
"num_shards": 11,
"tenant": "",
"zonegroup": "0fdaf584-33ee-4be7-978c-fb2771181aee",
"placement_rule": "default-placement",
"explicit_placement": {
"data_pool": "",
"data_extra_pool": "",
"index_pool": ""
},
"id": "76366a48-2ab8-4830-9cf7-f9b2fe432c5a.44325.2",
"marker": "76366a48-2ab8-4830-9cf7-f9b2fe432c5a.44325.2",
"index_type": "Normal",
"owner": "operator",
"ver": "0#1,1#1,2#1,3#1,4#1,5#1,6#1,7#1,8#1,9#1,10#1",
"master_ver": "0#0,1#0,2#0,3#0,4#0,5#0,6#0,7#0,8#0,9#0,10#0",
"mtime": "2025-08-25T10:12:11.164221Z",
"creation_time": "2025-08-25T10:09:12.571418Z",
"max_marker": "0#,1#,2#,3#,4#,5#,6#,7#,8#,9#,10#",
"usage": {},
"bucket_quota": {
"enabled": true,
"check_on_raw": false,
"max_size": -1,
"max_size_kb": 0,
"max_objects": 1024         
}
}

禁用 bucket 配额

bash 复制代码
[root@ceph1 ~]# radosgw-admin quota set --quota-scope=bucket --bucket=webapp --max-objects=-1
[root@ceph1 ~]# radosgw-admin quota disable --quota-scope=bucket --bucket=webapp
global 级别
bash 复制代码
[root@ceph1 ~]# radosgw-admin global quota set --quota-scope user --max-objects 
2048
Global quota changes saved. Use 'period update' to apply them to the staging 
period, and 'period commit' to commit the new period.
{
    "user quota": {
        "enabled": false,
        "check_on_raw": false,
        "max_size": -1,
        "max_size_kb": 0,
        "max_objects": 2048
    }
}
[root@ceph1 ~]# radosgw-admin global quota enable --quota-scope user
Global quota changes saved. Use 'period update' to apply them to the staging 
period, and 'period commit' to commit the new period.
{
"user quota": {
"enabled": true,
"check_on_raw": false,
"max_size": -1,
"max_size_kb": 0,
"max_objects": 2048
}
}
# 查看配额
[root@ceph1 ~]# radosgw-admin global quota get --quota-scope user
{
"user quota": {
"enabled": true,
"check_on_raw": false,
"max_size": -1,
"max_size_kb": 0,
"max_objects": 2048
}
}
[root@ceph1 ~]# radosgw-admin period update --commit

针对所有bucket配额

bash 复制代码
[root@ceph1 ~]# radosgw-admin global quota set --quota-scope bucket --max-objects 
2048
Global quota changes saved. Use 'period update' to apply them to the staging 
period, and 'period commit' to commit the new period.
{
"bucket quota": {
"enabled": false,
"check_on_raw": false,
"max_size": -1,
"max_size_kb": 0,
"max_objects": 2048
}
}
[root@ceph1 ~]# radosgw-admin global quota enable --quota-scope bucket
Global quota changes saved. Use 'period update' to apply them to the staging 
period, and 'period commit' to commit the new period.
{
"bucket quota": {
"enabled": true,
"check_on_raw": false,
"max_size": -1,
"max_size_kb": 0,
"max_objects": 2048
}
}
# 查看配额
[root@ceph1 ~]# radosgw-admin global quota get --quota-scope bucket
{
"bucket quota": {
"enabled": true,
"check_on_raw": false,
"max_size": -1,
"max_size_kb": 0,
"max_objects": 2048
}
}
[root@ceph1 ~]# radosgw-admin period update --commit
相关推荐
qq_356408663 小时前
Kubernetes Loki 日志收集系统部署文档 (读写分离模式 + Ceph S3 + Nginx 日志分离)
ceph·nginx·kubernetes
潮起鲸落入海5 小时前
ceph分布式存储认证和授权,块存储管理
分布式·ceph
AOwhisky2 天前
Ceph系列第三期:Ceph 集群核心配置与管理
linux·运维·数据库·笔记·ceph
AOwhisky2 天前
Ceph系列第五期:Ceph 对象存储(RADOS Gateway)精讲
linux·运维·笔记·ceph·gateway·对象存储
AOwhisky2 天前
Ceph系列第四期:Ceph块存储(RBD)精讲
linux·运维·笔记·ceph·云计算·rbd
三十..4 天前
Ceph分布式存储核心技术精要与运维实践指南
运维·分布式·ceph
一个行走的民4 天前
Ceph OSD NUMA 亲和性、Page Cache 跨 NUMA 访问与绑核实践
ceph
潮起鲸落入海4 天前
ceph集群组件管理 ceph orch 和ceph config命令
ceph
bukeyiwanshui4 天前
20260529 Ceph 分布式存储 认证和授权管理
ceph