基于华为云ECS平台的分布式存储系统Ceph-部署指导

实验简介
任务一：Ceph集群部署
- [步骤0 初始化设置](#步骤0 初始化设置)
- [步骤1 集群部署](#步骤1 集群部署)
- [步骤2 添加 OSD 服务](#步骤2 添加 OSD 服务)
- [步骤 3 添加 mgr 服务](#步骤 3 添加 mgr 服务)
- [步骤4 统一集群配置](#步骤4 统一集群配置)
- [步骤5 添加mds服务](#步骤5 添加mds服务)
任务二：Ceph集群运用
- [步骤1 文件系统运用](#步骤1 文件系统运用)
- [步骤2 块存储运用](#步骤2 块存储运用)
任务三：Ceph常用命令操作
- [步骤1 服务相关](#步骤1 服务相关)
- [步骤2 查看](#步骤2 查看)
- [步骤3 rbd 相关操作](#步骤3 rbd 相关操作)
- [步骤4 DOS 相关](#步骤4 DOS 相关)
- [步骤 5 清除 Ceph 相关配置](#步骤 5 清除 Ceph 相关配置)
实验小结

实验简介

实验背景

Ceph 是一个开源的、统一的分布式存储系统，设计初衷是提供较好的性能、可靠性和可扩展性。其中"统一"是指 Ceph 可以一套存储系统同时提供块设备存储、文件系统存储和对象存储三种存储功能。

实验环境

本次实验采用华为公有云 ECS（CentOS 7.9）作为实验平台，远程连接工具使用 MobaXterm。

实验目的

掌握 Ceph 集群部署
掌握 Ceph 文件系统及块存储的运用
掌握 Ceph 常用命令使用

实验准备

虚拟机 4 台（CentOS 7.9），此处 IP 地址为举例使用，实际实验过程中请使用管理员分配的 IP。

服务器	IP	配置规格	角色
ceph01（管理节点）	192.168.56.11	CPU：2 核内存：4G 磁盘：1 块系统盘，2 块 Ceph 存储盘	admin、osd、mon、mgr
ceph02（管理节点）	192.168.56.12	CPU：2 核内存：4G 磁盘：1 块系统盘，2 块 Ceph 存储盘	osd、mds、mon、mgr
ceph03（管理节点）	192.168.56.13	CPU：2 核内存：4G 磁盘：1 块系统盘，2 块 Ceph 存储盘	osd、mds、mon、mgr
client	192.168.56.10	CPU：1 核内存：2G	client

任务一：Ceph集群部署

步骤0 初始化设置

如无特别说明，以下操作需在四台主机（ceph01、ceph02、ceph03、client）上分别执行。

设置主机名

由于新购 ECS 实例的默认主机名可能与实验规划不一致，需手动设置主机名，以确保后续集群配置中主机名解析、SSH 互信及服务识别正常。请根据主机角色分别执行以下命令：

bash 复制代码

# 在 ceph01 上执行
hostnamectl set-hostname ceph01

# 在 ceph02 上执行
hostnamectl set-hostname ceph02

# 在 ceph03 上执行
hostnamectl set-hostname ceph03

# 在 client 上执行
hostnamectl set-hostname client

设置完成后，可通过 hostname 命令确认当前主机名是否生效。

配置 SSH 会话保活

为避免实验过程中 SSH 会话因长时间不操作被弹出断开，建议在四台主机上分别执行以下命令，追加 SSH 保活配置：

bash 复制代码

cat >> /etc/ssh/sshd_config <<'EOF'
TCPKeepAlive yes
ClientAliveInterval 60
ClientAliveCountMax 3
EOF

配置完成后，重启 sshd 服务使参数生效：

bash 复制代码

systemctl restart sshd

说明：上述配置表示每 60 秒向客户端发送一次保活探测，连续 3 次无响应后才断开连接，可有效维持实验过程中的远程会话稳定性。

步骤1 集群部署

申请实验资源

自行创建主机或从服务器管理员处获取实验资源，注意：IP 地址可能差异

需要的主机信息参考实验简介之实验准备。

关闭 SELinux 和 Firewalld，添加主机名解析

四台主机上都需要执行

使用远程工具登录至四台服务器。

关闭 SELinux，修改配置文件：/etc/selinux/config，将 SELINUX=permissive 修改为：SELINUX=disabled，保存后退出并重启主机。

添加主机名解析，vi /etc/hosts，添加以下解析内容，保存退出（下列 IP 为实例 IP，实验操作中以实际 IP 为准）：

bash 复制代码

192.168.56.11  ceph01
192.168.56.12  ceph02
192.168.56.13  ceph03
192.168.56.10  client

配置YUM源

配置基础 YUM 源（使用华为云 CentOS 7 镜像）：

bash 复制代码

curl -o /etc/yum.repos.d/CentOS-Base.repo https://repo.huaweicloud.com/repository/conf/CentOS-7-reg.repo
yum clean all
yum makecache

添加 Ceph 软件源（使用阿里云镜像）：

bash 复制代码

cat > /etc/yum.repos.d/ceph.repo <<EOF
[ceph]
name=ceph
baseurl=http://mirrors.aliyun.com/ceph/rpm-luminous/el7/x86_64/
gpgcheck=0

[ceph-noarch]
name=ceph-noarch
baseurl=http://mirrors.aliyun.com/ceph/rpm-luminous/el7/noarch/
gpgcheck=0
EOF

集群所有节点安装 epel-release：

bash 复制代码

yum -y install epel-release yum-plugin-priorities yum-utils ntpdate vim

配置时间同步，安装 chrony 服务：

bash 复制代码

yum -y install chrony

修改配置文件 vim /etc/chrony.conf，添加如下内容，保存退出：

text 复制代码

server ntp1.aliyun.com iburst
server ntp2.aliyun.com iburst
server ntp3.aliyun.com iburst

重启 chronyd 服务：

bash 复制代码

systemctl enable chronyd && systemctl restart chronyd

Ceph集群部署安装

如无特别说明，均在管理节点ceph01上执行

配置 ceph01 到其它节点的 SSH 免密码登录，输入：

bash 复制代码

ssh-keygen

输出：（过程中按3次回车确认）

text 复制代码

Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:9XjMY+m/arqpDUF80viC05jHiKwPo+CIp+y1mldyOg4 root@ceph01
The key's randomart image is:
+---[RSA 2048]----+
|                 |
|       . o       |
|        = +      |
|   . . O = = .   |
|    o * S o O    |
|   o o o o + .   |
|.E+.=   .   .    |
|*.=B.    o ...   |
|*O+oo   ..=+..o. |
+----[SHA256]-----+

将本地公钥复制到其它主机，输入：

bash 复制代码

for i in ceph01 ceph02 ceph03 client ; do ssh-copy-id $i ; done

输出：

bash 复制代码

/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
The authenticity of host 'ceph01 (192.168.56.11)' can't be established.
ECDSA key fingerprint is SHA256:gfrmf93G4d1wiTzqmfeLk4jvHZehhX5/MYZEtNhJWCE.
ECDSA key fingerprint is MD5:d5:c7:89:8f:ad:2c:62:08:c4:40:8d:6c:54:7f:9e:e0.
Are you sure you want to continue connecting (yes/no)? yes   # 输入yes
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
root@ceph01's password:                                     #  输入密码

Number of key(s) added: 1

Now try logging into the machine, with:   "ssh 'ceph01'"
and check to make sure that only the key(s) you wanted were added.

/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
The authenticity of host 'ceph02 (192.168.56.12)' can't be established.
ECDSA key fingerprint is SHA256:gfrmf93G4d1wiTzqmfeLk4jvHZehhX5/MYZEtNhJWCE.
ECDSA key fingerprint is MD5:d5:c7:89:8f:ad:2c:62:08:c4:40:8d:6c:54:7f:9e:e0.
Are you sure you want to continue connecting (yes/no)? yes      # 输入yes
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
root@ceph02's password:                                         #  输入密码

Number of key(s) added: 1

Now try logging into the machine, with:   "ssh 'ceph02'"
and check to make sure that only the key(s) you wanted were added.

/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
The authenticity of host 'ceph03 (192.168.56.13)' can't be established.
ECDSA key fingerprint is SHA256:gfrmf93G4d1wiTzqmfeLk4jvHZehhX5/MYZEtNhJWCE.
ECDSA key fingerprint is MD5:d5:c7:89:8f:ad:2c:62:08:c4:40:8d:6c:54:7f:9e:e0.
Are you sure you want to continue connecting (yes/no)? yes       # 输入yes
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
root@ceph03's password:                                          #  输入密码

Number of key(s) added: 1

Now try logging into the machine, with:   "ssh 'ceph03'"
and check to make sure that only the key(s) you wanted were added.

/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
The authenticity of host 'ceph03 (192.168.56.10)' can't be established.
ECDSA key fingerprint is SHA256:gfrmf93G4d1wiTzqmfeLk4jvHZehhX5/MYZEtNhJWCE.
ECDSA key fingerprint is MD5:d5:c7:89:8f:ad:2c:62:08:c4:40:8d:6c:54:7f:9e:e0.
Are you sure you want to continue connecting (yes/no)? yes       # 输入yes
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
root@ceph03's password:                                          #  输入密码

Number of key(s) added: 1

Now try logging into the machine, with:   "ssh 'client'"
and check to make sure that only the key(s) you wanted were added.

安装 ceph-deploy 相关工具（集群 3 个节点都执行）：

bash 复制代码

yum install -y ceph-deploy ceph ceph-radosgw snappy leveldb gdisk python-argparse gperftools-libs

创建新的集群：

bash 复制代码

cd /etc/ceph/
ceph-deploy new ceph01 ceph02 ceph03

执行完毕后，可以看到 /etc/ceph 目录中生成了三个文件：

ceph.conf 为 Ceph 配置文件，ceph-deploy-ceph.log 为 ceph-deploy 日志文件，ceph.mon.keyring 为 Ceph Monitor 的密钥环。

修改 ceph.conf 配置文件，增加副本数为 2：

bash 复制代码

echo "osd_pool_default_size = 2" >> ceph.conf

初始化 MON 节点并收集所有密钥：

bash 复制代码

ceph-deploy mon create-initial

使用命令查看集群状态是否正常：

bash 复制代码

ceph -s

步骤2 添加 OSD 服务

如无特别说明，均在管理节点ceph01上执行

Ceph 自动分区

接下来需要创建 OSD 了，OSD 是最终数据存储的地方，这里我们准备了3个 OSD 节点。官方建议为 OSD 及其日志使用独立硬盘或分区作为存储空间，也可以使用目录的方式创建。注意：系统盘 /dev/vda 已被操作系统占用，不可用于存储数据，因此本实验使用第二块磁盘 /dev/vdb 作为 OSD 数据盘。依次输入：

bash 复制代码

ceph-deploy disk zap ceph01 /dev/vdb
ceph-deploy disk zap ceph02 /dev/vdb
ceph-deploy disk zap ceph03 /dev/vdb

输出：

text 复制代码

[ceph01][DEBUG ] connected to host: ceph01
[ceph01][DEBUG ] detect platform information from remote host
[ceph01][DEBUG ] detect machine type
[ceph01][DEBUG ] find the location of an executable
[ceph_deploy.osd][INFO  ] Distro info: CentOS Linux 7.5.1804 Core
[ceph01][DEBUG ] zeroing last few blocks of device
[ceph01][DEBUG ] find the location of an executable
[ceph01][INFO  ] Running command: /usr/sbin/ceph-volume lvm zap /dev/vdb
[ceph01][WARNIN] --> Zapping: /dev/vdb
[ceph01][WARNIN] --> --destroy was not specified, but zapping a whole device will remove the partition table
[ceph01][WARNIN] Running command: dd if=/dev/zero of=/dev/vdb bs=1M count=10
[ceph01][WARNIN]  stderr: 10+0 records in
[ceph01][WARNIN] 10+0 records out
[ceph01][WARNIN] 10485760 bytes (10 MB) copied
[ceph01][WARNIN]  stderr: , 0.0574561 s, 183 MB/s
[ceph01][WARNIN] --> Zapping successful for: <Raw Device: /dev/vdb>

添加 OSD 节点

依次输入：

bash 复制代码

ceph-deploy osd create ceph01 --data /dev/vdb
ceph-deploy osd create ceph02 --data /dev/vdb
ceph-deploy osd create ceph03 --data /dev/vdb

输出：

text 复制代码

[ceph01][WARNIN] Running command: systemctl start ceph-osd@0
[ceph01][WARNIN] --> ceph-volume lvm activate successful for osd ID: 0
[ceph01][WARNIN] --> ceph-volume lvm create successful for: /dev/vdb
[ceph01][INFO  ] checking OSD status...
[ceph01][DEBUG ] find the location of an executable
[ceph01][INFO  ] Running command: /bin/ceph --cluster=ceph osd stat --format=json
[ceph_deploy.osd][DEBUG ] Host ceph01 is now ready for osd use.

查看 OSD 节点的状态

输入：

bash 复制代码

ceph-deploy osd list ceph01 ceph02 ceph03

输出：

text 复制代码

[ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf
[ceph_deploy.cli][INFO  ] Invoked (2.0.1): /usr/bin/ceph-deploy osd list ceph01 ceph02 ceph03
[ceph_deploy.cli][INFO  ] ceph-deploy options:
[ceph_deploy.cli][INFO  ]  username                      : None
[ceph_deploy.cli][INFO  ]  verbose                       : False
[ceph_deploy.cli][INFO  ]  debug                         : False
[ceph_deploy.cli][INFO  ]  overwrite_conf                : False
[ceph_deploy.cli][INFO  ]  subcommand                    : list
[ceph_deploy.cli][INFO  ]  quiet                         : False
[ceph_deploy.cli][INFO  ]  cd_conf                       : <ceph_deploy.conf.cephdeploy.Conf instance at 0x7f0ae5bdd680>
[ceph_deploy.cli][INFO  ]  cluster                       : ceph
[ceph_deploy.cli][INFO  ]  host                          : ['ceph01', 'ceph02', 'ceph03']
[ceph_deploy.cli][INFO  ]  func                          : <function osd at 0x7f0ae5e2e9b0>
[ceph_deploy.cli][INFO  ]  ceph_conf                     : None
[ceph_deploy.cli][INFO  ]  default_release               : False
[ceph01][DEBUG ] connected to host: ceph01
[ceph01][DEBUG ] detect platform information from remote host
[ceph01][DEBUG ] detect machine type
[ceph01][DEBUG ] find the location of an executable
[ceph_deploy.osd][INFO  ] Distro info: CentOS Linux 7.5.1804 Core
[ceph_deploy.osd][DEBUG ] Listing disks on ceph01...
[ceph01][DEBUG ] find the location of an executable
[ceph01][INFO  ] Running command: /usr/sbin/ceph-volume lvm list
[ceph01][DEBUG ]
[ceph01][DEBUG ]
[ceph01][DEBUG ] ====== osd.0 =======
[ceph01][DEBUG ]
[ceph01][DEBUG ]   [block]    /dev/ceph-7b294ab0-69e5-4a30-a3f4-ce915dbdda32/osd-block-fa694403-2f98-4f05-87f2-b68eb1eab8b7
[ceph01][DEBUG ]
[ceph01][DEBUG ]       type                      block
[ceph01][DEBUG ]       osd id                    0
[ceph01][DEBUG ]       cluster fsid              bcba353b-b630-4046-a614-a47a673dda71
[ceph01][DEBUG ]       cluster name              ceph
[ceph01][DEBUG ]       osd fsid                  fa694403-2f98-4f05-87f2-b68eb1eab8b7
[ceph01][DEBUG ]       encrypted                 0
[ceph01][DEBUG ]       cephx lockbox secret
[ceph01][DEBUG ]       block uuid                R1mXB7-f4hB-c6r8-TqFp-Afwr-drOn-l9c0va
[ceph01][DEBUG ]       block device              /dev/ceph-7b294ab0-69e5-4a30-a3f4-ce915dbdda32/osd-block-fa694403-2f98-4f05-87f2-b68eb1eab8b7
[ceph01][DEBUG ]       vdo                       0
[ceph01][DEBUG ]       crush device class        None
[ceph01][DEBUG ]       devices                   /dev/vdb
[ceph02][DEBUG ] connected to host: ceph02
[ceph02][DEBUG ] detect platform information from remote host
[ceph02][DEBUG ] detect machine type
[ceph02][DEBUG ] find the location of an executable
[ceph_deploy.osd][INFO  ] Distro info: CentOS Linux 7.5.1804 Core
[ceph_deploy.osd][DEBUG ] Listing disks on ceph02...
[ceph02][DEBUG ] find the location of an executable
[ceph02][INFO  ] Running command: /usr/sbin/ceph-volume lvm list
[ceph02][DEBUG ]
[ceph02][DEBUG ]
[ceph02][DEBUG ] ====== osd.1 =======
[ceph02][DEBUG ]
[ceph02][DEBUG ]   [block]    /dev/ceph-4291509f-a524-4ac7-b0b1-6e120fdf7996/osd-block-b4de1fa8-7828-47d6-b74f-df209630a9b5
[ceph02][DEBUG ]
[ceph02][DEBUG ]       type                      block
[ceph02][DEBUG ]       osd id                    1
[ceph02][DEBUG ]       cluster fsid              bcba353b-b630-4046-a614-a47a673dda71
[ceph02][DEBUG ]       cluster name              ceph
[ceph02][DEBUG ]       osd fsid                  b4de1fa8-7828-47d6-b74f-df209630a9b5
[ceph02][DEBUG ]       encrypted                 0
[ceph02][DEBUG ]       cephx lockbox secret
[ceph02][DEBUG ]       block uuid                ivWiMi-fWBW-K9Hd-oTL5-f2Nk-Zm0x-xUUV5h
[ceph02][DEBUG ]       block device              /dev/ceph-4291509f-a524-4ac7-b0b1-6e120fdf7996/osd-block-b4de1fa8-7828-47d6-b74f-df209630a9b5
[ceph02][DEBUG ]       vdo                       0
[ceph02][DEBUG ]       crush device class        None
[ceph02][DEBUG ]       devices                   /dev/vdb
[ceph03][DEBUG ] connected to host: ceph03
[ceph03][DEBUG ] detect platform information from remote host
[ceph03][DEBUG ] detect machine type
[ceph03][DEBUG ] find the location of an executable
[ceph_deploy.osd][INFO  ] Distro info: CentOS Linux 7.5.1804 Core
[ceph_deploy.osd][DEBUG ] Listing disks on ceph03...
[ceph03][DEBUG ] find the location of an executable
[ceph03][INFO  ] Running command: /usr/sbin/ceph-volume lvm list
[ceph03][DEBUG ]
[ceph03][DEBUG ]
[ceph03][DEBUG ] ====== osd.2 =======
[ceph03][DEBUG ]
[ceph03][DEBUG ]   [block]    /dev/ceph-3195d959-28e1-4c15-b0d3-bb99c681d82c/osd-block-0502bd91-73be-4c31-a9bf-97d1fcf5066e
[ceph03][DEBUG ]
[ceph03][DEBUG ]       type                      block
[ceph03][DEBUG ]       osd id                    2
[ceph03][DEBUG ]       cluster fsid              bcba353b-b630-4046-a614-a47a673dda71
[ceph03][DEBUG ]       cluster name              ceph
[ceph03][DEBUG ]       osd fsid                  0502bd91-73be-4c31-a9bf-97d1fcf5066e
[ceph03][DEBUG ]       encrypted                 0
[ceph03][DEBUG ]       cephx lockbox secret
[ceph03][DEBUG ]       block uuid                ROCLd3-6I42-LGt0-jOlm-jvgU-oCWq-ufKgAf
[ceph03][DEBUG ]       block device              /dev/ceph-3195d959-28e1-4c15-b0d3-bb99c681d82c/osd-block-0502bd91-73be-4c31-a9bf-97d1fcf5066e
[ceph03][DEBUG ]       vdo                       0
[ceph03][DEBUG ]       crush device class        None
[ceph03][DEBUG ]       devices                   /dev/vdb

步骤 3 添加 mgr 服务

输入：

bash 复制代码

ceph-deploy mgr create ceph01 ceph02 ceph03

输出：

text 复制代码

[ceph03][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph03][WARNIN] mgr keyring does not exist yet, creating one
[ceph03][DEBUG ] create a keyring file
[ceph03][DEBUG ] create path recursively if it doesn't exist
[ceph03][INFO  ] Running command: ceph --cluster ceph --name client.bootstrap-mgr --keyring /var/lib/ceph/bootstrap-mgr/ceph.keyring auth get-or-create mgr.ceph03 mon allow profile mgr osd allow * mds allow * -o /var/lib/ceph/mgr/ceph-ceph03/keyring
[ceph03][INFO  ] Running command: systemctl enable ceph-mgr@ceph03
[ceph03][WARNIN] Created symlink from /etc/systemd/system/ceph-mgr.target.wants/ceph-mgr@ceph03.service to /usr/lib/systemd/system/ceph-mgr@.service.
[ceph03][INFO  ] Running command: systemctl start ceph-mgr@ceph03
[ceph03][INFO  ] Running command: systemctl enable ceph.target

步骤4 统一集群配置

用 ceph-deploy 把配置文件和 admin 密钥拷贝到管理节点和 Ceph 节点，这样你每次执行 Ceph 命令行时就无需指定 monitor 地址和 ceph.client.admin.keyring 了。

输入：

bash 复制代码

ceph-deploy admin ceph01 ceph02 ceph03

输出：

text 复制代码

[ceph_deploy.admin][DEBUG ] Pushing admin keys and conf to ceph01
[ceph01][DEBUG ] connected to host: ceph01
[ceph01][DEBUG ] detect platform information from remote host
[ceph01][DEBUG ] detect machine type
[ceph01][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph_deploy.admin][DEBUG ] Pushing admin keys and conf to ceph02
[ceph02][DEBUG ] connected to host: ceph02
[ceph02][DEBUG ] detect platform information from remote host
[ceph02][DEBUG ] detect machine type
[ceph02][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph_deploy.admin][DEBUG ] Pushing admin keys and conf to ceph03
[ceph03][DEBUG ] connected to host: ceph03
[ceph03][DEBUG ] detect platform information from remote host
[ceph03][DEBUG ] detect machine type
[ceph03][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf

集群每一个节点授权（三个节点都执行）：

bash 复制代码

[root@ceph01 ceph]# chmod +r /etc/ceph/ceph.client.admin.keyring
[root@ceph02 ceph]# chmod +r /etc/ceph/ceph.client.admin.keyring
[root@ceph03 ceph]# chmod +r /etc/ceph/ceph.client.admin.keyring

步骤5 添加mds服务

MDS 是 Ceph 集群中的元数据服务器，通常它都不是必须的，只有在使用 CephFS 的时候才需要它，目前在云计算中用的更广泛的是另外两种存储方式。

MDS 虽然是元数据服务器，但是它不负责存储元数据，元数据也是被切成对象存在各个 OSD 节点中。

在创建 CephFS 时，要至少创建两个 Pool，一个用于存放数据，另一个用于存放元数据。Mds只是负责接受用户的元数据查询请求，然后从osd中把数据取出来映射进自己的内存中供客户访问。所以mds其实类似一个代理缓存服务器，替osd分担了用户的访问压力。

管理节点ceph01输入：

bash 复制代码

ceph-deploy mds create ceph02 ceph03

输出：

text 复制代码

[ceph_deploy.mds][DEBUG ] remote host will use systemd
[ceph_deploy.mds][DEBUG ] deploying mds bootstrap to ceph03
[ceph03][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph03][WARNIN] mds keyring does not exist yet, creating one
[ceph03][DEBUG ] create a keyring file
[ceph03][DEBUG ] create path if it doesn't exist
[ceph03][INFO  ] Running command: ceph --cluster ceph --name client.bootstrap-mds --keyring /var/lib/ceph/bootstrap-mds/ceph.keyring auth get-or-create mds.ceph03 osd allow rwx mds allow mon allow profile mds -o /var/lib/ceph/mds/ceph-ceph03/keyring
[ceph03][INFO  ] Running command: systemctl enable ceph-mds@ceph03
[ceph03][WARNIN] Created symlink from /etc/systemd/system/ceph-mds.target.wants/ceph-mds@ceph03.service to /usr/lib/systemd/system/ceph-mds@.service.
[ceph03][INFO  ] Running command: systemctl start ceph-mds@ceph03
[ceph03][INFO  ] Running command: systemctl enable ceph.target

任务二：Ceph集群运用

步骤1 文件系统运用

创建文件系统存储池

查看文件系统，管理节点 ceph01 输入：

bash 复制代码

ceph fs ls

输出：

text 复制代码

No filesystems enabled

创建存储池。

语法说明：

text 复制代码

ceph osd pool create cephfs_data <pg_num>
ceph osd pool create cephfs_metadata <pg_num>

其中：<pg_num> = 128。

关于创建存储池：

确定 pg_num 取值是强制性的，因为不能自动计算。下面是几个常用的值：

少于 5 个 OSD 时可把 pg_num 设置为 128
OSD 数量在 5 到 10 个时，可把 pg_num 设置为 512
OSD 数量在 10 到 50 个时，可把 pg_num 设置为 4096
OSD 数量大于 50 时，你得理解权衡方法、以及如何自己计算 pg_num 取值
自己计算 pg_num 取值时可借助 pgcalc 工具

text 复制代码

https://ceph.com/pgcalc/

CephFS 需要两个 Pools：cephfs-data 和 cephfs-metadata，分别存储文件数据和文件元数据。

输入：

bash 复制代码

ceph osd pool create ceph_data 128

输出：

text 复制代码

pool 'ceph_data' created

输入：

bash 复制代码

ceph osd pool create ceph_metadata 128

输出：

text 复制代码

pool 'ceph_metadata' created

创建文件系统

创建好存储池后，就可以用 fs new 命令创建文件系统。

命令：

text 复制代码

ceph fs new <fs_name> cephfs_metadata cephfs_data

其中：<fs_name> = cephfs 可自定义。

给创建的 2 个存储池创建文件系统，输入：

bash 复制代码

ceph fs new cephfs ceph_metadata ceph_data

输出：

text 复制代码

new fs with metadata pool 2 and data pool 1

查看文件系统，输入：

bash 复制代码

ceph fs ls

输出：

text 复制代码

name: cephfs, metadata pool: ceph_metadata, data pools: [ceph_data ]

查看 MDS 的状态，输入：

bash 复制代码

ceph fs status cephfs

输出：

text 复制代码

new fs with metadata pool 2 and data pool 1
[root@ceph01 ceph]# ceph fs ls
name: cephfs, metadata pool: ceph_metadata, data pools: [ceph_data ]
[root@ceph01 ceph]# ceph fs status cephfs
cephfs - 0 clients
======
+------+--------+--------+---------------+-------+-------+
| Rank | State  |  MDS   |    Activity   |  dns  |  inos |
+------+--------+--------+---------------+-------+-------+
|  0   | active | ceph03 | Reqs:    0 /s |   10  |   12  |
+------+--------+--------+---------------+-------+-------+
+---------------+----------+-------+-------+
|      Pool     |   type   |  used | avail |
+---------------+----------+-------+-------+
| ceph_metadata | metadata | 2246  | 26.9G |
|   ceph_data   |   data   |    0  | 26.9G |
+---------------+----------+-------+-------+

+-------------+
| Standby MDS |
+-------------+
|    ceph02   |
+-------------+
MDS version: ceph version 12.2.13 (584a20eb0237c657dc0567da126be145106a47e) luminous (stable)

挂载客户端

要挂载 Ceph 文件系统，如果知道监视器 IP 地址可以用 mount 命令、或者用 mount.ceph 工具来自动解析监视器 IP 地址。此处演示用内核驱动的方式挂载文件系统。

在client创建挂载点，在client 节点输入：

bash 复制代码

mkdir /data

查看密钥，在ceph01节点输入：

bash 复制代码

cat /etc/ceph/ceph.client.admin.keyring

输出：

text 复制代码

[client.admin]
        key = AQCBXHBkLaiCKhAAzSbc4YPJQZWzhSOaLN5GGQ==

使用秘钥挂载，在client节点输入（此处以实际环境中ceph01的IP和密钥为准）：

bash 复制代码

mount -t ceph 192.168.56.11:6789:/ /data/ -o name=admin,secret=AQCBXHBkLaiCKhAAzSbc4YPJQZWzhSOaLN5GGQ==

查看挂载结果，在client节点输入：

bash 复制代码

df -hT

输出：

text 复制代码

[root@client ~]# df -hT
Filesystem                Type      Size  Used Avail Use% Mounted on
/dev/mapper/centos-root   xfs       150G  1.1G  149G   1% /
devtmpfs                  devtmpfs  3.9G     0  3.9G   0% /dev
tmpfs                     tmpfs     3.9G     0  3.9G   0% /dev/shm
tmpfs                     tmpfs     3.9G  8.6M  3.9G   1% /run
tmpfs                     tmpfs     3.9G     0  3.9G   0% /sys/fs/cgroup
/dev/sda2                 xfs       197M  120M   77M  61% /boot
/dev/mapper/centos-ddhome xfs       350G   33M  350G   1% /ddhome
tmpfs                     tmpfs     783M     0  783M   0% /run/user/0
192.168.56.11:6789:/      ceph       27G     0   27G   0% /data

取消挂载，在client节点输入：

bash 复制代码

umount /data/

步骤2 块存储运用

Ceph 支持一个非常好的特性，以 COW（写时复制 copy-on-write）的方式从 RBD 快照创建克隆，在 Ceph 中被称为快照分层。分层特性允许用户创建多个 Ceph RBD 克隆实例。这些特性应用于 OpenStack 等云平台中，使用快照形式保护 Ceph RBD 镜像，快照是只读的，但 COW 克隆是完全可以写，可以多次来孵化实例，对云平台来说是非常有用。

创建块存储的存储池

查看内核是否支持使用 RBD，输入（在 client 执行）：

bash 复制代码

modprobe rbd

如果有错误信息说明内核不支持，需要先去升级一下内核。

输入：

bash 复制代码

lsmod | grep rbd

输出：

text 复制代码

[root@client ~]# lsmod | grep rbd
rbd                    83728  0
libceph               301687  2 rbd,ceph

创建存储池，输入（ceph01 执行）：

bash 复制代码

ceph osd pool create rbd 64

输出：

text 复制代码

pool 'rbd' created

查看当前的PG值，输入：

bash 复制代码

ceph osd pool get rbd pg_num

输出：

text 复制代码

pg_num: 64

创建块设备镜像

要想把块设备加入某节点，得先在 Ceph 存储集群中创建一个镜像，使用下列命令：

text 复制代码

#rbd create --size {megabytes} {pool-name}/{image-name} -m mon节点的ip地址

如果创建镜像时不指定存储池，它将使用默认的 rbd 存储池。

输入（以下在 ceph01 执行）：

bash 复制代码

rbd create --size 102400 rbd/test1

查看 rbd 的信息：

bash 复制代码

rbd ls

输出：

text 复制代码

test1

查看镜像的信息：

bash 复制代码

rbd info test1

输出：

text 复制代码

rbd image 'test1':
        size 100GiB in 25600 objects
        order 22 (4MiB objects)
        block_name_prefix: rbd_data.10616b8b4567
        format: 2
        features: layering, exclusive-lock, object-map, fast-diff, deep-flatten
        flags:
        create_timestamp: Sat May 27 15:00:27 20XX

在client上执行

映射块设备

客户端确定安装ceph-common包（在client执行）：

bash 复制代码

yum -y install ceph-common
yum -y install ceph-common

将ceph01上的ceph配置文件拷贝到client（在ceph01上操作）：

bash 复制代码

scp /etc/ceph/* client:/etc/ceph/

以下在client上输入：

bash 复制代码

[root@client ~]# rbd feature disable test1 object-map fast-diff deep-flatten exclusive-lock
 
[root@client ~]# rbd map test1
/dev/rbd0
[root@client ~]# ls /dev/rbd0 
/dev/rbd0

可以看见在/dev下创建了一个叫rbd0的设备文件。

挂载使用

挂载rbd的Linux 服务器首先需要机器支持ceph客户端，如果是一台新机器的话，请安装ceph，然后同步下配置文件。

创建挂载点：

bash 复制代码

root@client ~]# mkdir /cephrbd

格式化磁盘：

bash 复制代码

[root@client ~]# mkfs.xfs -f /dev/rbd0
meta-data=/dev/rbd0              isize=512    agcount=16, agsize=1638400 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=1        finobt=0, sparse=0
data     =                       bsize=4096   blocks=26214400, imaxpct=25
         =                       sunit=1024   swidth=1024 blks
naming   =version 2              bsize=4096   ascii-ci=0 ftype=1
log      =internal log           bsize=4096   blocks=12800, version=2
         =                       sectsz=512   sunit=8 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

挂载：

bash 复制代码

mount /dev/rbd0 /cephrbd/

查看挂载磁盘：

bash 复制代码

[root@client ~]# df -hT
Filesystem                Type      Size  Used Avail Use% Mounted on
/dev/mapper/centos-root   xfs       150G  1.3G  149G   1% /
devtmpfs                  devtmpfs  3.9G     0  3.9G   0% /dev
tmpfs                     tmpfs     3.9G     0  3.9G   0% /dev/shm
tmpfs                     tmpfs     3.9G  8.7M  3.9G   1% /run
tmpfs                     tmpfs     3.9G     0  3.9G   0% /sys/fs/cgroup
/dev/sda2                 xfs       197M  120M   77M  61% /boot
/dev/mapper/centos-ddhome xfs       350G   33M  350G   1% /ddhome
tmpfs                     tmpfs     783M     0  783M   0% /run/user/0
/dev/rbd0                 xfs       100G   33M  100G   1% /cephrbd

测试写入数据：

bash 复制代码

[root@client ~]# dd if=/dev/zero of=/cephrbd/file bs=100M count=1
1+0 records in
1+0 records out
104857600 bytes (105 MB) copied, 0.326624 s, 321 MB/s
[root@client ~]#

创建快照并测试回滚

按以下步骤创建快照并验证回滚功能：

bash 复制代码

# 1. 挂载 RBD（如果尚未映射，先 rbd map）
rbd map rbd/test1
mount /dev/rbd0 /cephrbd

# 2. 创建测试文件
echo "hello ceph" > /cephrbd/ceph.txt

# 3. 确保数据刷到磁盘（重要）
sync

# 4. 创建快照
rbd snap create rbd/test1@mysnap

# 5. 删除文件
rm -f /cephrbd/ceph.txt
sync

# 6. 离开挂载点，然后卸载
cd /
umount /cephrbd

# 7. 确认已卸载
mount | grep cephrbd   # 应无输出
df -h | grep cephrbd   # 应无输出

# 8. 回滚快照
rbd snap rollback rbd/test1@mysnap

# 9. 重新挂载
mount /dev/rbd0 /cephrbd

# 10. 验证文件恢复
ls -l /cephrbd/ceph.txt
cat /cephrbd/ceph.txt

执行完上述步骤后，应能看到 ceph.txt 文件已恢复，且内容为 hello ceph。

模板与克隆

查看块设备的format（format必须为2）：

bash 复制代码

[root@client ~]# rbd info rbd/test1
rbd image 'test1':
        size 102400 MB in 25600 objects
        order 22 (4096 kB objects)
        block_name_prefix: rbd_data.10616b8b4567
        format: 2
        features: layering
        flags:
        
如果不是format 2 可以创建时指定
# rbd create rbd/test1 --size 102400 --image-format 2

创建克隆前，把快照保存起来，不然会出错：

bash 复制代码

[root@client ~]# rbd snap protect rbd/test1@mysnap

可以使用rbd snap unprotect rbd/test1@mysnap去掉这个保护，但是这样就不能克隆了。

卸载挂载点：

bash 复制代码

[root@client ~]# umount /dev/rbd0

克隆设备：

bash 复制代码

[root@client ~]# rbd clone rbd/test1@mysnap rbd/test2

查看rbd，输入：

bash 复制代码

[root@client ~]# rbd ls

输出：

text 复制代码

test1
test2

任务三：Ceph常用命令操作

步骤1 服务相关

查看所有服务

bash 复制代码

systemctl status ceph\*.service ceph\*.target

输出：

text 复制代码

[root@ceph01 ~]# systemctl status ceph\*.service ceph\*.target
● ceph-mon@ceph01.service - Ceph cluster monitor daemon
   Loaded: loaded (/usr/lib/systemd/system/ceph-mon@.service; enabled; vendor preset: disabled)
   Active: active (running) since Fri 20XX-XX-XX-26 15:14:47 CST; 1 day 3h ago
 Main PID: 2046 (ceph-mon)
   CGroup: /system.slice/system-ceph\x2dmon.slice/ceph-mon@ceph01.service
           └─2046 /usr/bin/ceph-mon -f --cluster ceph --id ceph01 --setuser ceph --setgroup ceph

May 26 15:14:47 ceph01 systemd[1]: Started Ceph cluster monitor daemon.
May 26 15:14:47 ceph01 systemd[1]: Starting Ceph cluster monitor daemon...
May 26 15:32:57 ceph01 ceph-mon[2046]: 20XX-XX-XX-26 15:32:57.485985 7f8bd8ead700 -1 log_channel(cluster) log [ERR] : He..._DOWN)
May 27 03:48:01 ceph01 ceph-mon[2046]: 20XX-XX-XX-27 03:48:01.474315 7f8bd3ea3700 -1 received  signal: Hangup from  PID:...UID: 0
May 27 03:48:01 ceph01 ceph-mon[2046]: 20XX-XX-XX-27 03:48:01.478454 7f8bd3ea3700 -1 received  signal: Hangup from  PID:...UID: 0

● ceph.target - ceph target allowing to start/stop all ceph*@.service instances at once
   Loaded: loaded (/usr/lib/systemd/system/ceph.target; enabled; vendor preset: enabled)
   Active: active since Fri 20XX-XX-XX-26 15:13:11 CST; 1 day 3h ago

May 26 15:13:11 ceph01 systemd[1]: Reached target ceph target allowing to start/stop all ceph*@.service instances at once.
May 26 15:13:11 ceph01 systemd[1]: Starting ceph target allowing to start/stop all ceph*@.service instances at once.

● ceph-mgr@ceph01.service - Ceph cluster manager daemon
   Loaded: loaded (/usr/lib/systemd/system/ceph-mgr@.service; enabled; vendor preset: disabled)
   Active: active (running) since Fri 20XX-XX-XX-26 15:52:36 CST; 1 day 2h ago
 Main PID: 12999 (ceph-mgr)
   CGroup: /system.slice/system-ceph\x2dmgr.slice/ceph-mgr@ceph01.service
           └─12999 /usr/bin/ceph-mgr -f --cluster ceph --id ceph01 --setuser ceph --setgroup ceph

May 26 15:52:36 ceph01 systemd[1]: Started Ceph cluster manager daemon.
May 26 15:52:36 ceph01 systemd[1]: Starting Ceph cluster manager daemon...
May 27 03:48:01 ceph01 ceph-mgr[12999]: 20XX-XX-XX-27 03:48:01.474585 7f289852e700 -1 received  signal: Hangup from  PID...UID: 0
May 27 03:48:01 ceph01 ceph-mgr[12999]: 20XX-XX-XX-27 03:48:01.478571 7f289852e700 -1 received  signal: Hangup from  PID...UID: 0

● ceph-mgr.target - ceph target allowing to start/stop all ceph-mgr@.service instances at once
   Loaded: loaded (/usr/lib/systemd/system/ceph-mgr.target; enabled; vendor preset: enabled)
   Active: active since Fri 20XX-XX-XX-26 15:13:26 CST; 1 day 3h ago

May 26 15:13:26 ceph01 systemd[1]: Reached target ceph target allowing to start/stop all ceph-mgr@.service instances at once.
May 26 15:13:26 ceph01 systemd[1]: Starting ceph target allowing to start/stop all ceph-mgr@.service instances at once.

● ceph-mon.target - ceph target allowing to start/stop all ceph-mon@.service instances at once
   Loaded: loaded (/usr/lib/systemd/system/ceph-mon.target; enabled; vendor preset: enabled)
   Active: active since Fri 20XX-XX-XX-26 15:13:24 CST; 1 day 3h ago

May 26 15:13:24 ceph01 systemd[1]: Reached target ceph target allowing to start/stop all ceph-mon@.service instances at once.
May 26 15:13:24 ceph01 systemd[1]: Starting ceph target allowing to start/stop all ceph-mon@.service instances at once.

● ceph-osd.target - ceph target allowing to start/stop all ceph-osd@.service instances at once
   Loaded: loaded (/usr/lib/systemd/system/ceph-osd.target; enabled; vendor preset: enabled)
   Active: active since Fri 20XX-XX-XX-26 15:13:29 CST; 1 day 3h ago

May 26 15:13:29 ceph01 systemd[1]: Reached target ceph target allowing to start/stop all ceph-osd@.service instances at once.
May 26 15:13:29 ceph01 systemd[1]: Starting ceph target allowing to start/stop all ceph-osd@.service instances at once.

● ceph-osd@0.service - Ceph object storage daemon osd.0
   Loaded: loaded (/usr/lib/systemd/system/ceph-osd@.service; enabled-runtime; vendor preset: disabled)
   Active: active (running) since Fri 20XX-XX-XX-26 15:33:00 CST; 1 day 3h ago
 Main PID: 12781 (ceph-osd)
   CGroup: /system.slice/system-ceph\x2dosd.slice/ceph-osd@0.service
           └─12781 /usr/bin/ceph-osd -f --cluster ceph --id 0 --setuser ceph --setgroup ceph

May 26 15:33:00 ceph01 systemd[1]: Starting Ceph object storage daemon osd.0...
May 26 15:33:00 ceph01 systemd[1]: Started Ceph object storage daemon osd.0.
May 26 15:33:00 ceph01 ceph-osd[12781]: starting osd.0 at - osd_data /var/lib/ceph/osd/ceph-0 /var/lib/ceph/osd/ceph-0/journal
May 26 15:33:00 ceph01 ceph-osd[12781]: 20XX-XX-XX-26 15:33:00.718443 7f9fe0cebec0 -1 osd.0 0 log_to_monitors {default=true}
May 26 15:33:02 ceph01 ceph-osd[12781]: 20XX-XX-XX-26 15:33:02.168030 7f9fc70c1700 -1 osd.0 0 waiting for initial osdmap
May 27 03:48:01 ceph01 ceph-osd[12781]: 20XX-XX-XX-27 03:48:01.474591 7f9fbd0ad700 -1 received  signal: Hangup from  PID...UID: 0
May 27 03:48:01 ceph01 ceph-osd[12781]: 20XX-XX-XX-27 03:48:01.478523 7f9fbd0ad700 -1 received  signal: Hangup from  PID...UID: 0

● ceph-radosgw.target - ceph target allowing to start/stop all ceph-radosgw@.service instances at once
   Loaded: loaded (/usr/lib/systemd/system/ceph-radosgw.target; enabled; vendor preset: enabled)
   Active: active since Fri 20XX-XX-XX-26 15:13:30 CST; 1 day 3h ago

May 26 15:13:30 ceph01 systemd[1]: Reached target ceph target allowing to start/stop all ceph-radosgw@.service insta...t once.
May 26 15:13:30 ceph01 systemd[1]: Starting ceph target allowing to start/stop all ceph-radosgw@.service instances at once.

● ceph-mds.target - ceph target allowing to start/stop all ceph-mds@.service instances at once
   Loaded: loaded (/usr/lib/systemd/system/ceph-mds.target; enabled; vendor preset: enabled)
   Active: active since Fri 20XX-XX-XX-26 15:13:25 CST; 1 day 3h ago

May 26 15:13:25 ceph01 systemd[1]: Reached target ceph target allowing to start/stop all ceph-mds@.service instances at once.
May 26 15:13:25 ceph01 systemd[1]: Starting ceph target allowing to start/stop all ceph-mds@.service instances at once.
Hint: Some lines were ellipsized, use -l to show in full.
[root@ceph01 ~]#

关闭所有服务

bash 复制代码

systemctl stop ceph\*.service ceph\*.target

启动服务

bash 复制代码

systemctl start ceph.target

关闭所有 OSD 服务

bash 复制代码

systemctl stop ceph-osd\*.service

关闭所有 MON 服务

bash 复制代码

systemctl stop ceph-mon\*.service

启动单个 MON 服务

bash 复制代码

systemctl start ceph-mon@{hostname}

启动单个 MDS 服务

bash 复制代码

systemctl start ceph-mds@{hostname}

步骤2 查看

查看命令帮助

输入：

bash 复制代码

ceph --help

输出：

text 复制代码

[root@ceph01 ~]# ceph --help

 General usage:
 ==============
usage: ceph [-h] [-c CEPHCONF] [-i INPUT_FILE] [-o OUTPUT_FILE]
            [--setuser SETUSER] [--setgroup SETGROUP] [--id CLIENT_ID]
            [--name CLIENT_NAME] [--cluster CLUSTER]
            [--admin-daemon ADMIN_SOCKET] [-s] [-w] [--watch-debug]
            [--watch-info] [--watch-sec] [--watch-warn] [--watch-error]
            [--watch-channel WATCH_CHANNEL] [--version] [--verbose]
            [--concise] [-f {json,json-pretty,xml,xml-pretty,plain}]
            [--connect-timeout CLUSTER_TIMEOUT]

Ceph administration tool

optional arguments:
  -h, --help            request mon help
  -c CEPHCONF, --conf CEPHCONF
                        ceph configuration file
  -i INPUT_FILE, --in-file INPUT_FILE
                        input file, or "-" for stdin
  -o OUTPUT_FILE, --out-file OUTPUT_FILE
                        output file, or "-" for stdout
  --setuser SETUSER     set user file permission
  --setgroup SETGROUP   set group file permission
  --id CLIENT_ID, --user CLIENT_ID
                        client id for authentication
  --name CLIENT_NAME, -n CLIENT_NAME
                        client name for authentication
  --cluster CLUSTER     cluster name
  --admin-daemon ADMIN_SOCKET
                        submit admin-socket commands ("help" for help
  -s, --status          show cluster status
  -w, --watch           watch live cluster changes
  --watch-debug         watch debug events
  --watch-info          watch info events
  --watch-sec           watch security events
  --watch-warn          watch warn events
  --watch-error         watch error events
  --watch-channel WATCH_CHANNEL
......

查看状态

输入：

bash 复制代码

ceph -s

输出：

text 复制代码

[root@ceph01 ~]# ceph -s
  cluster:
    id:     bcba353b-b630-4046-a614-a47a673dda71
    health: HEALTH_WARN
            1 osds down
            1 host (1 osds) down
            Degraded data redundancy: 85/272 objects degraded (31.250%), 36 pgs degraded, 209 pgs undersized
            application not enabled on 1 pool(s)

  services:
    mon: 3 daemons, quorum ceph01,ceph02,ceph03
    mgr: ceph02(active), standbys: ceph03, ceph01
    mds: cephfs-1/1/1 up  {0=ceph03=up:active}, 1 up:standby
    osd: 3 osds: 2 up, 3 in

  data:
    pools:   3 pools, 320 pgs
    objects: 136 objects, 309MiB
    usage:   6.31GiB used, 53.7GiB / 60.0GiB avail
    pgs:     85/272 objects degraded (31.250%)
             173 active+undersized
             111 active+clean
             36  active+undersized+degraded

查看所有的 OSD 池

输入：

bash 复制代码

ceph osd pool ls

输出：

text 复制代码

[root@ceph01 ~]# ceph osd pool ls
ceph_data
ceph_metadata
rbd
[root@ceph01 ~]#

查看存储池使用情况

输入：

bash 复制代码

rados df

输出：

text 复制代码

[root@ceph01 ~]#  rados df
POOL_NAME     USED    OBJECTS CLONES COPIES MISSING_ON_PRIMARY UNFOUND DEGRADED RD_OPS RD      WR_OPS WR
ceph_data          0B       0      0      0                  0       0        0      0      0B      0     0B
ceph_metadata 2.52KiB      21      0     42                  0       0        0      0      0B     48  12KiB
rbd            309MiB     115     55    230                  0       0        0   2580 10.1MiB   1378 360MiB

total_objects    136
total_used       5.40GiB
total_avail      34.6GiB
total_space      40.0GiB
[root@ceph01 ~]#

步骤3 rbd 相关操作

在 rbd pool 下创建 image

输入：

bash 复制代码

rbd create image1 --size 60G

默认在 rbd pool 下创建一个名为 image1，大小为 60G 的 image，等同于：

bash 复制代码

rbd create rbd/image1 --size 60G --image-format 2

列出所有的块设备 image

输入：

bash 复制代码

rbd list

输出：

text 复制代码

[root@ceph01 ~]# rbd list
image1
test1

查看某个具体的 image 的信息

输入：

bash 复制代码

rbd info image1

输出：

text 复制代码

[root@ceph01 ~]# rbd info image1
rbd image 'image1':
        size 60GiB in 15360 objects
        order 22 (4MiB objects)
        block_name_prefix: rbd_data.5e2f6b8b4567
        format: 2
        features: layering, exclusive-lock, object-map, fast-diff, deep-flatten
        flags:
        create_timestamp: Mon May 29 09:47:42 20XX
[root@ceph01 ~]#

关掉image1的一些feature

输入：

bash 复制代码

rbd feature disable image1 exclusive-lock, object-map, fast-diff, deep-flatten

把test_image块设备映射到操作系统

输入：

bash 复制代码

rbd map image1

输出：

text 复制代码

[root@ceph01 ~]# rbd map image1
/dev/rbd0

显示已经映射的块设备

输入：

bash 复制代码

rbd showmapped

输出：

text 复制代码

[root@ceph01 ~]# rbd showmapped
id pool image  snap device
0  rbd  image1 -    /dev/rbd0

取消映射

输入：

bash 复制代码

rbd unmap image1

删除一个rbd image

输入：

bash 复制代码

rbd rm image1

输出：

text 复制代码

[root@ceph01 ~]# rbd rm image1
Removing image: 100% complete...done.

步骤4 DOS 相关

查看 Ceph 集群的 pool

输入：

bash 复制代码

rados lspools

输出：

text 复制代码

[root@ceph01 ~]# rados lspools
ceph_data
ceph_metadata
rbd

创建名为的 Ceph pool

输入：

bash 复制代码

rados mkpool test1

输出：

text 复制代码

[root@ceph01 ~]# rados mkpool test1
successfully created pool test1

步骤 5 清除 Ceph 相关配置

停止所有进程

输入：

bash 复制代码

systemctl stop ceph\*.service ceph\*.target

卸载所有 Ceph 程序

输入：

bash 复制代码

ceph-deploy uninstall [{ceph-node}]

此处 ${ceph-node}$ 为变量，可以代表 ceph01、ceph02 和 ceph03

输出：

text 复制代码

[root@ceph01 ~]# ceph-deploy  uninstall  ceph03
[ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf
[ceph_deploy.cli][INFO  ] Invoked (2.0.1): /usr/bin/ceph-deploy uninstall ceph03
[ceph_deploy.cli][INFO  ] ceph-deploy options:
[ceph_deploy.cli][INFO  ]  username                      : None
[ceph_deploy.cli][INFO  ]  verbose                       : False
[ceph_deploy.cli][INFO  ]  overwrite_conf                : False
[ceph_deploy.cli][INFO  ]  quiet                         : False
[ceph_deploy.cli][INFO  ]  cd_conf                       : <ceph_deploy.conf.cephdeploy.Conf instance at 0x7f0aaa59ae60>
[ceph_deploy.cli][INFO  ]  cluster                       : ceph
[ceph_deploy.cli][INFO  ]  host                          : ['ceph03']
[ceph_deploy.cli][INFO  ]  func                          : <function uninstall at 0x7f0aaae37938>
[ceph_deploy.cli][INFO  ]  ceph_conf                     : None
[ceph_deploy.cli][INFO  ]  default_release               : False
[ceph_deploy.install][INFO  ] note that some dependencies *will not* be removed because they can cause issues with qemu-kvm
[ceph_deploy.install][INFO  ] like: librbd1 and librados2
[ceph_deploy.install][DEBUG ] Uninstalling on cluster ceph hosts ceph03
[ceph_deploy.install][DEBUG ] Detecting platform for host ceph03 ...
[ceph03][DEBUG ] connected to host: ceph03
[ceph03][DEBUG ] detect platform information from remote host
[ceph03][DEBUG ] detect machine type
[ceph_deploy.install][INFO  ] Distro info: CentOS Linux 7.5.1804 Core
[ceph03][INFO  ] Uninstalling Ceph on ceph03
[ceph03][INFO  ] Running command: yum -y -q remove ceph ceph-release ceph-common ceph-radosgw
[ceph03][WARNIN] No Match for argument: ceph-release
[ceph03][INFO  ] Running command: yum clean all
[ceph03][DEBUG ] Loaded plugins: fastestmirror, priorities
[ceph03][DEBUG ] Cleaning repos: base ceph ceph-noarch epel extras updates
[ceph03][DEBUG ] Cleaning up everything
[ceph03][DEBUG ] Maybe you want: rm -rf /var/cache/yum, to also free up space taken by orphaned data from disabled or removed repos
[ceph03][DEBUG ] Cleaning up list of fastest mirrors
[root@ceph01 ~]#

删除 Ceph 相关的安装包

输入：

bash 复制代码

ceph-deploy purge {ceph-node} [{ceph-data}]

输出：

text 复制代码

[root@ceph01 ~]# ceph-deploy purge ceph03
[ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf
[ceph_deploy.cli][INFO  ] Invoked (2.0.1): /usr/bin/ceph-deploy purge ceph03
[ceph_deploy.cli][INFO  ] ceph-deploy options:
[ceph_deploy.cli][INFO  ]  username                      : None
[ceph_deploy.cli][INFO  ]  verbose                       : False
[ceph_deploy.cli][INFO  ]  debug                         : False
[ceph_deploy.cli][INFO  ]  overwrite_conf                : False
[ceph_deploy.cli][INFO  ]  quiet                         : False
[ceph_deploy.cli][INFO  ]  cd_conf                       : <ceph_deploy.conf.cephdeploy.Conf instance at 0x7fbb101cab90>
[ceph_deploy.cli][INFO  ]  cluster                       : ceph
[ceph_deploy.cli][INFO  ]  host                          : ['ceph03']
[ceph_deploy.cli][INFO  ]  func                          : <function purge at 0x7fbb10a619b0>
[ceph_deploy.cli][INFO  ]  ceph_conf                     : None
[ceph_deploy.cli][INFO  ]  default_release               : False
[ceph_deploy.install][INFO  ] note that some dependencies *will not* be removed because they can cause issues with qemu-kvm
[ceph_deploy.install][INFO  ] like: librbd1 and librados2
[ceph_deploy.install][DEBUG ] Purging on cluster ceph hosts ceph03
[ceph_deploy.install][DEBUG ] Detecting platform for host ceph03 ...
[ceph03][DEBUG ] connected to host: ceph03
[ceph03][DEBUG ] detect platform information from remote host
[ceph03][DEBUG ] detect machine type
[ceph_deploy.install][INFO  ] Distro info: CentOS Linux 7.5.1804 Core
[ceph03][INFO  ] Purging Ceph on ceph03
[ceph03][INFO  ] Running command: yum -y -q remove ceph ceph-release ceph-common ceph-radosgw
[ceph03][WARNIN] No Match for argument: ceph
[ceph03][WARNIN] No Match for argument: ceph-release
[ceph03][WARNIN] No Match for argument: ceph-common
[ceph03][WARNIN] No Match for argument: ceph-radosgw
[ceph03][INFO  ] Running command: yum clean all
[ceph03][DEBUG ] Loaded plugins: fastestmirror, priorities
[ceph03][DEBUG ] Cleaning repos: base ceph ceph-noarch epel extras updates
[ceph03][DEBUG ] Cleaning up everything
[ceph03][DEBUG ] Maybe you want: rm -rf /var/cache/yum, to also free up space taken by orphaned data from disabled or removed repos
[root@ceph01 ~]#

删除 Ceph 相关的配置

输入：

bash 复制代码

ceph-deploy purgedata {ceph-node} [{ceph-data}]

输出：

text 复制代码

[root@ceph01 ~]# ceph-deploy purgedata ceph03
[ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf
[ceph_deploy.cli][INFO  ] Invoked (2.0.1): /usr/bin/ceph-deploy purgedata ceph03
[ceph_deploy.cli][INFO  ] ceph-deploy options:
[ceph_deploy.cli][INFO  ]  username                      : None
[ceph_deploy.cli][INFO  ]  verbose                       : False
[ceph_deploy.cli][INFO  ]  overwrite_conf                : False
[ceph_deploy.cli][INFO  ]  quiet                         : False
[ceph_deploy.cli][INFO  ]  cd_conf                       : <ceph_deploy.conf.cephdeploy.Conf instance at 0x7f64876ee518>
[ceph_deploy.cli][INFO  ]  cluster                       : ceph
[ceph_deploy.cli][INFO  ]  host                          : ['ceph03']
[ceph_deploy.cli][INFO  ]  func                          : <function purgedata at 0x7f6487f85a28>
[ceph_deploy.cli][INFO  ]  ceph_conf                     : None
[ceph_deploy.cli][INFO  ]  default_release               : False
[ceph_deploy.install][DEBUG ] Purging data from cluster ceph hosts ceph03
[ceph03][DEBUG ] connected to host: ceph03
[ceph03][DEBUG ] detect platform information from remote host
[ceph03][DEBUG ] detect machine type
[ceph03][DEBUG ] find the location of an executable
[ceph03][DEBUG ] connected to host: ceph03
[ceph03][DEBUG ] detect platform information from remote host
[ceph03][DEBUG ] detect machine type
[ceph_deploy.install][INFO  ] Distro info: CentOS Linux 7.5.1804 Core
[ceph03][INFO  ] purging data on ceph03
[ceph03][INFO  ] Running command: rm -rf --one-file-system -- /var/lib/ceph
[ceph03][WARNIN] OSDs may still be mounted, trying to unmount them
[ceph03][INFO  ] Running command: find /var/lib/ceph -mindepth 1 -maxdepth 2 -type d -exec umount {} ;
[ceph03][WARNIN] umount: /var/lib/ceph/osd: not mounted
[ceph03][INFO  ] Running command: rm -rf --one-file-system -- /var/lib/ceph
[ceph03][INFO  ] Running command: rm -rf --one-file-system -- /etc/ceph/
[root@ceph01 ~]#

删除key

输入：

bash 复制代码

ceph-deploy forgetkeys

输出：

text 复制代码

[root@ceph01 ~]# ceph-deploy forgetkeys
[ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf
[ceph_deploy.cli][INFO  ] Invoked (2.0.1): /usr/bin/ceph-deploy forgetkeys
[ceph_deploy.cli][INFO  ] ceph-deploy options:
[ceph_deploy.cli][INFO  ]  username                      : None
[ceph_deploy.cli][INFO  ]  verbose                       : False
[ceph_deploy.cli][INFO  ]  debug                         : False
[ceph_deploy.cli][INFO  ]  overwrite_conf                : False
[ceph_deploy.cli][INFO  ]  quiet                         : False
[ceph_deploy.cli][INFO  ]  cd_conf                       : <ceph_deploy.conf.cephdeploy.Conf instance at 0x7f50712e6248>
[ceph_deploy.cli][INFO  ]  cluster                       : ceph
[ceph_deploy.cli][INFO  ]  func                          : <function forgetkeys at 0x7f5071b7d398>
[ceph_deploy.cli][INFO  ]  ceph_conf                     : None
[ceph_deploy.cli][INFO  ]  default_release               : False
[root@ceph01 ~]#

卸载ceph-deploy管理

输入：

bash 复制代码

yum -y remove ceph-deploy

输出：

text 复制代码

[root@ceph01 ~]# yum -y remove ceph-deploy
Loaded plugins: fastestmirror, priorities
Resolving Dependencies
--> Running transaction check
---> Package ceph-deploy.noarch 0:2.0.1-0 will be erased
--> Finished Dependency Resolution

Dependencies Resolved

==============================================================================================================================
 Package                        Arch                      Version                       Repository                       Size
==============================================================================================================================
Removing:
 ceph-deploy                    noarch                    2.0.1-0                       @ceph-noarch                    1.2 M

Transaction Summary
==============================================================================================================================
Remove  1 Package

Installed size: 1.2 M
Downloading packages:
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Erasing    : ceph-deploy-2.0.1-0.noarch                                                                                 1/1
  Verifying  : ceph-deploy-2.0.1-0.noarch                                                                                 1/1

Removed:
  ceph-deploy.noarch 0:2.0.1-0

Complete!
[root@ceph01 ~]#