4. ceph存储使用流程

Martin_wjc2024-06-21 13:21

ceph存储使用流程

一、ceph三种存储接口

文件系统存储 cephFS, 依赖于MDS

块存储 RBD

对象存储 RGW

二、文件系统存储

1、在ceph集群中部署MDS

bash 复制代码 
[root@node01 ceph]# ceph-deploy mds create node01 node02 node03 
[ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf
[ceph_deploy.cli][INFO  ] Invoked (2.0.1): /usr/bin/ceph-deploy mds create node01 node02 node03
[ceph_deploy.cli][INFO  ] ceph-deploy options:
[ceph_deploy.cli][INFO  ]  username                      : None
[ceph_deploy.cli][INFO  ]  verbose                       : False
[ceph_deploy.cli][INFO  ]  overwrite_conf                : False
[ceph_deploy.cli][INFO  ]  subcommand                    : create
[ceph_deploy.cli][INFO  ]  quiet                         : False
[ceph_deploy.cli][INFO  ]  cd_conf                       : <ceph_deploy.conf.cephdeploy.Conf instance at 0x7fc36a1277a0>
[ceph_deploy.cli][INFO  ]  cluster                       : ceph
[ceph_deploy.cli][INFO  ]  func                          : <function mds at 0x7fc36a16c050>
[ceph_deploy.cli][INFO  ]  ceph_conf                     : None
[ceph_deploy.cli][INFO  ]  mds                           : [('node01', 'node01'), ('node02', 'node02'), ('node03', 'node03')]
[ceph_deploy.cli][INFO  ]  default_release               : False
[ceph_deploy.mds][DEBUG ] Deploying mds, cluster ceph hosts node01:node01 node02:node02 node03:node03
[node01][DEBUG ] connected to host: node01 
[node01][DEBUG ] detect platform information from remote host
[node01][DEBUG ] detect machine type
[ceph_deploy.mds][INFO  ] Distro info: CentOS Linux 7.9.2009 Core
[ceph_deploy.mds][DEBUG ] remote host will use systemd
[ceph_deploy.mds][DEBUG ] deploying mds bootstrap to node01
[node01][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[node01][WARNIN] mds keyring does not exist yet, creating one
[node01][DEBUG ] create a keyring file
[node01][DEBUG ] create path if it doesn't exist
[node01][INFO  ] Running command: ceph --cluster ceph --name client.bootstrap-mds --keyring /var/lib/ceph/bootstrap-mds/ceph.keyring auth get-or-create mds.node01 osd allow rwx mds allow mon allow profile mds -o /var/lib/ceph/mds/ceph-node01/keyring
[node01][INFO  ] Running command: systemctl enable ceph-mds@node01
[node01][WARNIN] Created symlink from /etc/systemd/system/ceph-mds.target.wants/ceph-mds@node01.service to /usr/lib/systemd/system/ceph-mds@.service.
[node01][INFO  ] Running command: systemctl start ceph-mds@node01
[node01][INFO  ] Running command: systemctl enable ceph.target
[node02][DEBUG ] connected to host: node02 
[node02][DEBUG ] detect platform information from remote host
[node02][DEBUG ] detect machine type
[ceph_deploy.mds][INFO  ] Distro info: CentOS Linux 7.9.2009 Core
[ceph_deploy.mds][DEBUG ] remote host will use systemd
[ceph_deploy.mds][DEBUG ] deploying mds bootstrap to node02
[node02][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[node02][WARNIN] mds keyring does not exist yet, creating one
[node02][DEBUG ] create a keyring file
[node02][DEBUG ] create path if it doesn't exist
[node02][INFO  ] Running command: ceph --cluster ceph --name client.bootstrap-mds --keyring /var/lib/ceph/bootstrap-mds/ceph.keyring auth get-or-create mds.node02 osd allow rwx mds allow mon allow profile mds -o /var/lib/ceph/mds/ceph-node02/keyring
[node02][INFO  ] Running command: systemctl enable ceph-mds@node02
[node02][WARNIN] Created symlink from /etc/systemd/system/ceph-mds.target.wants/ceph-mds@node02.service to /usr/lib/systemd/system/ceph-mds@.service.
[node02][INFO  ] Running command: systemctl start ceph-mds@node02
[node02][INFO  ] Running command: systemctl enable ceph.target
[node03][DEBUG ] connected to host: node03 
[node03][DEBUG ] detect platform information from remote host
[node03][DEBUG ] detect machine type
[ceph_deploy.mds][INFO  ] Distro info: CentOS Linux 7.9.2009 Core
[ceph_deploy.mds][DEBUG ] remote host will use systemd
[ceph_deploy.mds][DEBUG ] deploying mds bootstrap to node03
[node03][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[node03][WARNIN] mds keyring does not exist yet, creating one
[node03][DEBUG ] create a keyring file
[node03][DEBUG ] create path if it doesn't exist
[node03][INFO  ] Running command: ceph --cluster ceph --name client.bootstrap-mds --keyring /var/lib/ceph/bootstrap-mds/ceph.keyring auth get-or-create mds.node03 osd allow rwx mds allow mon allow profile mds -o /var/lib/ceph/mds/ceph-node03/keyring
[node03][INFO  ] Running command: systemctl enable ceph-mds@node03
[node03][WARNIN] Created symlink from /etc/systemd/system/ceph-mds.target.wants/ceph-mds@node03.service to /usr/lib/systemd/system/ceph-mds@.service.
[node03][INFO  ] Running command: systemctl start ceph-mds@node03
[node03][INFO  ] Running command: systemctl enable ceph.target
bash 复制代码 
[root@node01 ceph]# netstat -tunlp | grep mds
tcp        0      0 192.168.140.10:6805     0.0.0.0:*               LISTEN      2228/ceph-mds

2、创建存储池

bash 复制代码 
[root@node01 ceph]# ceph osd pool create db_data 128			// 存储实际数据
pool 'db_data' created

[root@node01 ceph]# ceph osd pool create db_metadata 64		// 存储元数据信息
pool 'db_metadata' created
[root@node01 ceph]# 
[root@node01 ceph]# ceph -s
  cluster:
    id:     e2010562-0bae-4999-9247-4017f875acc8
    health: HEALTH_OK
 
  services:
    mon: 3 daemons, quorum node01,node02,node03
    mgr: node01(active), standbys: node02, node03
    mds: mydata-1/1/1 up  {0=node01=up:active}, 2 up:standby
    osd: 3 osds: 3 up, 3 in
 
  data:
    pools:   2 pools, 192 pgs
    objects: 22  objects, 2.5 KiB
    usage:   3.0 GiB used, 57 GiB / 60 GiB avail
    pgs:     192 active+clean
bash 复制代码 
[root@node01 ceph]# ceph osd pool ls
db_data
db_metadata

说明:

一个文件系统存储需要两个RADOS存储池，一个用于存储实体数据，一个用于存储元数据

上面分别创建两个存储池，名称为db_data, db_metadata

分别指定存储池对应的PG数量为128， 64

存储池对应PG数量参考

少于5个OSD则PG数为128

5-10个OSD则PG数为512

10-50个OSD则PG数为1024

如果有更多的OSD需要自己理解计算

PG计算公式

Total PGs = ((Total_number_of_OSD * 100) / max_replication_count) / pool_count

3、创建文件系统存储

bash 复制代码 
[root@node01 ceph]# ceph fs new mydata db_metadata db_data 
new fs with metadata pool 2 and data pool 1

[root@node01 ceph]# ceph fs ls
name: mydata, metadata pool: db_metadata, data pools: [db_data ]

4、业务服务器挂载使用cephfs

ceph集群默认启用了cephx的认证，业务服务器要挂载使用ceph需要通过令牌认证

4.1 将认证的令牌导出，拷贝到业务服务器

bash 复制代码 
[root@node01 ceph]# cat ceph.client.admin.keyring 
[client.admin]
	key = AQBI5Gtm/aabIxAAc5MEuV05QyvjFNYhUJmnyA==
	caps mds = "allow *"
	caps mgr = "allow *"
	caps mon = "allow *"
	caps osd = "allow *"
[root@node01 ceph]# 

[root@node01 ceph]# ceph-authtool -p ceph.client.admin.keyring > /root/client.keyring
[root@node01 ceph]# 
[root@node01 ceph]# scp /root/client.keyring root@192.168.140.13:/root/
client.keyring                                                                                      100%   41    51.5KB/s   00:00    
[root@node01 ceph]#

4.2 业务服务器挂载使用ceph

bash 复制代码 
[root@app ~]# yum install -y ceph-fuse
bash 复制代码 
[root@app ~]# mount -t ceph node01:6789:/  /test1  -o name=admin,secretfile=/opt/admin.key

[root@app ~]# df -hT 
Filesystem              Type      Size  Used Avail Use% Mounted on
devtmpfs                devtmpfs  475M     0  475M   0% /dev
tmpfs                   tmpfs     487M     0  487M   0% /dev/shm
tmpfs                   tmpfs     487M  7.6M  479M   2% /run
tmpfs                   tmpfs     487M     0  487M   0% /sys/fs/cgroup
/dev/mapper/centos-root xfs        18G  1.8G   16G  11% /
/dev/sda1               xfs       497M  161M  336M  33% /boot
tmpfs                   tmpfs      98M     0   98M   0% /run/user/0
192.168.140.10:6789:/   ceph       18G     0   18G   0% /test1

5、删除文件系统存储

5.1 业务服务器取消挂载

5.2 修改ceph.conf，添加允许删除的配置；同步配置文件

bash 复制代码 
[root@node01 ceph]# tail -n 1 ceph.conf
mon_allow_pool_delete = true

[root@node01 ceph]# ceph-deploy --overwrite-conf admin node01 node02 node03

[root@node01 ceph]# systemctl restart ceph-mon.target

5.3 停掉所有集群节点的mds服务

bash 复制代码 
[root@node01 ceph]# systemctl stop ceph-mds.target

5.4 删除文件系统

bash 复制代码 
[root@node01 ceph]# ceph fs ls
name: mydata, metadata pool: db_metadata, data pools: [db_data ]

[root@node01 ceph]# ceph fs rm mydata --yes-i-really-mean-it

[root@node01 ceph]# ceph fs ls
No filesystems enabled

5.5 删除文件系统对应的存储池

bash 复制代码 
[root@node01 ceph]# ceph osd pool ls
db_data
db_metadata
[root@node01 ceph]# 
[root@node01 ceph]# 
[root@node01 ceph]# ceph osd pool delete db_metadata db_metadata --yes-i-really-really-mean-it
pool 'db_metadata' removed
[root@node01 ceph]# 
[root@node01 ceph]# ceph osd pool delete db_data db_data --yes-i-really-really-mean-it
pool 'db_data' removed

三、块存储的使用

1、将ceph的配置同步到业务服务器

bash 复制代码 
[root@node01 ceph]# ceph-deploy admin app
[ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf
[ceph_deploy.cli][INFO  ] Invoked (2.0.1): /usr/bin/ceph-deploy admin app
[ceph_deploy.cli][INFO  ] ceph-deploy options:
[ceph_deploy.cli][INFO  ]  username                      : None
[ceph_deploy.cli][INFO  ]  verbose                       : False
[ceph_deploy.cli][INFO  ]  overwrite_conf                : False
[ceph_deploy.cli][INFO  ]  quiet                         : False
[ceph_deploy.cli][INFO  ]  cd_conf                       : <ceph_deploy.conf.cephdeploy.Conf instance at 0x7f7567f307e8>
[ceph_deploy.cli][INFO  ]  cluster                       : ceph
[ceph_deploy.cli][INFO  ]  client                        : ['app']
[ceph_deploy.cli][INFO  ]  func                          : <function admin at 0x7f7568c46320>
[ceph_deploy.cli][INFO  ]  ceph_conf                     : None
[ceph_deploy.cli][INFO  ]  default_release               : False
[ceph_deploy.admin][DEBUG ] Pushing admin keys and conf to app
The authenticity of host 'app (192.168.140.13)' can't be established.
ECDSA key fingerprint is SHA256:bdjFNp2/Yyt+F2YvGN2prmrdzemgYD0YJ/CEFvGobN4.
ECDSA key fingerprint is MD5:3b:db:a0:e6:76:03:16:8f:8b:d7:cb:1f:d0:65:77:f0.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'app' (ECDSA) to the list of known hosts.
[app][DEBUG ] connected to host: app 
[app][DEBUG ] detect platform information from remote host
[app][DEBUG ] detect machine type
[app][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf

后续的操作均在业务服务器上执行

2、创建存储池、初始化

bash 复制代码 
[root@app ~]# ceph osd pool create block_pool 128
pool 'block_pool' created
[root@app ~]# 
[root@app ~]# rbd pool init block_pool

3、创建卷

bash 复制代码 
[root@app ~]# rbd create db_volume --pool block_pool --size 5000
[root@app ~]# 
[root@app ~]# rbd ls block_pool 
db_volume
[root@app ~]#

4、映射块设备

bash 复制代码 
[root@app ~]# rbd map block_pool/db_volume
rbd: sysfs write failed
RBD image feature set mismatch. You can disable features unsupported by the kernel with "rbd feature disable block_pool/db_volume object-map fast-diff deep-flatten".
In some cases useful info is found in syslog - try "dmesg | tail".
rbd: map failed: (6) No such device or address

[root@app ~]# rbd feature disable block_pool/db_volume object-map fast-diff deep-flatten

[root@app ~]# rbd map block_pool/db_volume
/dev/rbd0
[root@app ~]# lsblk 
NAME            MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sda               8:0    0   20G  0 disk 
├─sda1            8:1    0  500M  0 part /boot
└─sda2            8:2    0 19.5G  0 part 
  ├─centos-root 253:0    0 17.5G  0 lvm  /
  └─centos-swap 253:1    0    2G  0 lvm  [SWAP]
sr0              11:0    1  9.5G  0 rom  
rbd0            252:0    0  4.9G  0 disk

5、使用ceph的块设备存储数据

bash 复制代码 
[root@app ~]# mkfs -t xfs /dev/rbd0

[root@app ~]# mount /dev/rbd0 /test1/
[root@app ~]# df -hT | grep test1
/dev/rbd0               xfs       4.9G   33M  4.9G   1% /test1

[root@app ~]# touch /test1/{1..10}
[root@app ~]# ls /test1/
1  10  2  3  4  5  6  7  8  9

6、扩容

bash 复制代码 
[root@app ~]# rbd resize --size 8000 block_pool/db_volume 
Resizing image: 100% complete...done.

[root@app ~]# xfs_growfs /test1/

[root@app ~]# df -hT | grep test1
/dev/rbd0               xfs       7.9G   33M  7.8G   1% /test1

注意：xfs_growfs适用于xfs文件系统， ext4文件系统用resize2fs

7、缩容

必须处于卸载状态

xfs不支持直接缩容

bash 复制代码 
[root@app ~]# mkdir /backup
[root@app ~]# cp -ra /test1/* /backup/
[root@app ~]# 
[root@app ~]# umount /test1 
[root@app ~]# 
[root@app ~]# rbd resize --size 4000 block_pool/db_volume --allow-shrink
Resizing image: 100% complete...done.

[root@app ~]# mkfs -t xfs -f /dev/rbd0

[root@app ~]# mount /dev/rbd0 /test1/
[root@app ~]# cp -ra /backup/* /test1/
[root@app ~]# ls /test1/
1  10  2  3  4  5  6  7  8  9

8、删除块存储

bash 复制代码 
[root@app ~]# umount /test1 
[root@app ~]# rbd unmap /dev/rbd0
[root@app ~]# lsblk 
NAME            MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sda               8:0    0   20G  0 disk 
├─sda1            8:1    0  500M  0 part /boot
└─sda2            8:2    0 19.5G  0 part 
  ├─centos-root 253:0    0 17.5G  0 lvm  /
  └─centos-swap 253:1    0    2G  0 lvm  [SWAP]
sr0              11:0    1  9.5G  0 rom  

[root@app ~]# ceph osd pool delete block_pool block_pool --yes-i-really-really-mean-it 
pool 'block_pool' removed

四、对象存储

基于对象的存储， 每一个文件称为对象

每个文件存储后，对应惟一的下载地址

适用场景：非结构化数据(图片、视频、声频、动画)

对象存储依赖于rgw服务

1、创建rgw服务

bash 复制代码 
[root@node01 ceph]# ceph-deploy rgw create node01
[ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf
[ceph_deploy.cli][INFO  ] Invoked (2.0.1): /usr/bin/ceph-deploy rgw create node01
[ceph_deploy.cli][INFO  ] ceph-deploy options:
[ceph_deploy.cli][INFO  ]  username                      : None
[ceph_deploy.cli][INFO  ]  verbose                       : False
[ceph_deploy.cli][INFO  ]  rgw                           : [('node01', 'rgw.node01')]
[ceph_deploy.cli][INFO  ]  overwrite_conf                : False
[ceph_deploy.cli][INFO  ]  subcommand                    : create
[ceph_deploy.cli][INFO  ]  quiet                         : False
[ceph_deploy.cli][INFO  ]  cd_conf                       : <ceph_deploy.conf.cephdeploy.Conf instance at 0x7fa364d58fc8>
[ceph_deploy.cli][INFO  ]  cluster                       : ceph
[ceph_deploy.cli][INFO  ]  func                          : <function rgw at 0x7fa365a2d140>
[ceph_deploy.cli][INFO  ]  ceph_conf                     : None
[ceph_deploy.cli][INFO  ]  default_release               : False
[ceph_deploy.rgw][DEBUG ] Deploying rgw, cluster ceph hosts node01:rgw.node01
[node01][DEBUG ] connected to host: node01 
[node01][DEBUG ] detect platform information from remote host
[node01][DEBUG ] detect machine type
[ceph_deploy.rgw][INFO  ] Distro info: CentOS Linux 7.9.2009 Core
[ceph_deploy.rgw][DEBUG ] remote host will use systemd
[ceph_deploy.rgw][DEBUG ] deploying rgw bootstrap to node01
[node01][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[node01][WARNIN] rgw keyring does not exist yet, creating one
[node01][DEBUG ] create a keyring file
[node01][DEBUG ] create path recursively if it doesn't exist
[node01][INFO  ] Running command: ceph --cluster ceph --name client.bootstrap-rgw --keyring /var/lib/ceph/bootstrap-rgw/ceph.keyring auth get-or-create client.rgw.node01 osd allow rwx mon allow rw -o /var/lib/ceph/radosgw/ceph-rgw.node01/keyring
[node01][INFO  ] Running command: systemctl enable ceph-radosgw@rgw.node01
[node01][WARNIN] Created symlink from /etc/systemd/system/ceph-radosgw.target.wants/ceph-radosgw@rgw.node01.service to /usr/lib/systemd/system/ceph-radosgw@.service.
[node01][INFO  ] Running command: systemctl start ceph-radosgw@rgw.node01
[node01][INFO  ] Running command: systemctl enable ceph.target
[ceph_deploy.rgw][INFO  ] The Ceph Object Gateway (RGW) is now running on host node01 and default port 7480
bash 复制代码 
[root@node01 ceph]# ceph -s
  cluster:
    id:     e2010562-0bae-4999-9247-4017f875acc8
    health: HEALTH_OK
 
  services:
    mon: 3 daemons, quorum node01,node02,node03
    mgr: node01(active), standbys: node02, node03
    osd: 3 osds: 3 up, 3 in
    rgw: 1 daemon active
 
  data:
    pools:   4 pools, 32 pgs
    objects: 187  objects, 1.1 KiB
    usage:   3.0 GiB used, 57 GiB / 60 GiB avail
    pgs:     32 active+clean
 

[root@node01 ceph]# netstat -tunlp | grep 7480
tcp        0      0 0.0.0.0:7480            0.0.0.0:*               LISTEN      7409/radosgw

在客户端上测试对象存储的使用

2、安装s3cmd测试工具

bash 复制代码 
[root@app ~]# yum install -y s3cmd

3、生成连接对象存储需要的AK、SK

bash 复制代码 
[root@app ~]#  radosgw-admin user create --uid="testuser" --display-name="first user" | grep -E "access_key|secret_key"

4、创建.s3cfg配置文件，指定对象存储网关的连接信息

bash 复制代码 
[root@app ~]# cat .s3cfg 
[default]
access_key = AOFSKMEGLZKG8CY9B0NH
secret_key = VIQjjaxvdYHrYQv7Qv5QAI0z81cNf0oQCdi7EFxS
host_base = 192.168.140.10:7480
host_bucket = 192.168.140.10:7480/%(bucket)
cloudfront_host = 192.168.140.10:7480
use_https = False

5、创建桶

bash 复制代码 
[root@app ~]# s3cmd mb s3://test
Bucket 's3://test/' created
[root@app ~]# 
[root@app ~]# s3cmd ls
2024-06-17 07:42  s3://test

6、测试文件上传、下载

bash 复制代码 
[root@app ~]# s3cmd put /etc/fstab s3://test/fstab 
upload: '/etc/fstab' -> 's3://test/fstab'  [1 of 1]
 466 of 466   100% in    1s   360.49 B/s  done
bash 复制代码 
[root@app ~]# s3cmd get s3://test/fstab
download: 's3://test/fstab' -> './fstab'  [1 of 1]
 466 of 466   100% in    0s    11.24 KB/s  done
相关推荐
大数据编程之光4 分钟前
Flink Standalone集群模式安装部署全攻略
java·大数据·开发语言·面试·flink
B站计算机毕业设计超人6 分钟前
计算机毕业设计SparkStreaming+Kafka旅游推荐系统 旅游景点客流量预测 旅游可视化 旅游大数据 Hive数据仓库 机器学习 深度学习
大数据·数据仓库·hadoop·python·kafka·课程设计·数据可视化
在下不上天2 小时前
Flume日志采集系统的部署,实现flume负载均衡，flume故障恢复
大数据·开发语言·python
智慧化智能化数字化方案2 小时前
华为IPD流程管理体系L1至L5最佳实践-解读
大数据·华为
PersistJiao3 小时前
在 Spark RDD 中，sortBy 和 top 算子的各自适用场景
大数据·spark·top·sortby
2301_811274314 小时前
大数据基于Spring Boot的化妆品推荐系统的设计与实现
大数据·spring boot·后端
Yz98764 小时前
hive的存储格式
大数据·数据库·数据仓库·hive·hadoop·数据库开发
青云交4 小时前
大数据新视界 -- 大数据大厂之 Hive 数据导入：多源数据集成的策略与实战（上）（3/ 30）
大数据·数据清洗·电商数据·数据整合·hive 数据导入·多源数据·影视娱乐数据
武子康4 小时前
大数据-230 离线数仓 - ODS层的构建 Hive处理 UDF 与 SerDe 处理 与 当前总结
java·大数据·数据仓库·hive·hadoop·sql·hdfs
武子康4 小时前
大数据-231 离线数仓 - DWS 层、ADS 层的创建 Hive 执行脚本
java·大数据·数据仓库·hive·hadoop·mysql
热门推荐
01(欧拉)openEuler系统添加网卡文件配置流程、(欧拉)openEuler系统手动配置ipv6地址流程、(欧拉)openEuler系统网络管理说明02PyTorch机器学习实现液态神经网络03【HarmonyOS】HUAWEI DevEco Studio 下载地址汇总04玄机平台应急响应—webshell查杀05Coze扣子平台完整体验和实践（附国内和国际版对比）06Ubuntu 20.04使用Livox mid 360 测试 FAST_LIO07【目标跟踪】相机运动补偿08Unity中PICO实现 隔空取物 和 接触抓取物体09RAG 实践- Ollama+RagFlow 部署本地知识库10ICLR24大模型提示(1/11) | BadChain：大型语言模型的后门思维链提示