ceph部署方式有很多
官网安装地址
root@gc-test:/home/ceph# curl --silent --remote-name https://raw.githubusercontent.com/ceph/ceph/quincy/src/cephadm/cephadm
root@gc-test:/home/ceph# chmod +x cephadm
root@gc-test:/home/ceph# mv cephadm /usr/local/bin/
root@gc-test:/home/ceph# cephadm add-repo --release quincy
Installing repo GPG key from https://download.ceph.com/keys/release.gpg...
Installing repo file at /etc/apt/sources.list.d/ceph.list...
Updating package list...
Completed adding repo.
root@gc-test:/home/ceph# cephadm install ceph-common
Installing packages ['ceph-common']...下面就查看一下服务器的ip
如何查找ip呢，用ip a命令
你会看到类似：
2: ens33: <UP,BROADCAST> ...
inet 10.0.0.189/24 brd 10.0.0.255 scope global ens33
其中：
网卡名称：ens33、ens160、eth0 等
IP 地址：10.0.0.189（你需要的就是它）你应该把这个 IP 填给 bootstrap：
cephadm bootstrap --mon-ip 10.0.0.189注意，还得安装好docker，openssh-sever,如果没有安装好的话，就会出现问题
安装ssh服务
# 安装 SSH 服务
sudo apt-get update
sudo apt-get install -y openssh-server
# 启动并设置开机自启
sudo systemctl start ssh
sudo systemctl enable ssh
# 验证 SSH 服务状态（显示 active (running) 即正常）
sudo systemctl status ssh允许 root 用户 SSH 登录（cephadm 依赖 root 免密 SSH）
在文件/etc/ssh/sshd_config
添加或者修改
PermitRootLogin yes # 允许 root SSH 登录（默认是 prohibit-password/no）
PasswordAuthentication yes # 允许密码登录（临时方便测试，后续可改回密钥）然后重启ssh服务
sudo systemctl restart ssh验证本机 SSH 连通性（确保能连）
# 测试本机 SSH 连接（用 root 用户）
ssh root@192.168.244.128
# 首次连接输入 yes 确认指纹，然后输入 root 密码，能登录即成功如果显示密码不对，可能得先为root设置密码
node1@ubuntu:~/Desktop$ sudo passwd root
New password:
Retype new password:
passwd: password updated successfully重新执行 cephadm bootstrap 命令
# 停止 ceph 相关容器（如有）
sudo docker stop $(sudo docker ps -q --filter "name=ceph-*") 2>/dev/null
sudo docker rm $(sudo docker ps -aq --filter "name=ceph-*") 2>/dev/null
# 重新执行 bootstrap
sudo cephadm bootstrap --mon-ip 192.168.244.128然后可能会出现
niuniu@niuniu-virtual-machine:~/Desktop/ceph$ sudo cephadm bootstrap --mon-ip 192.168.244.128
ERROR: /etc/ceph/ceph.conf already exists; delete or pass --allow-overwrite to overwrite这个时候就需要
sudo cephadm bootstrap --mon-ip 192.168.244.128 --allow-overwrite
root@gc-test:/home/ceph# cephadm bootstrap --mon-ip 10.0.0.189
Verifying podman|docker is present...
Verifying lvm2 is present...
Verifying time synchronization is in place...
Unit chrony.service is enabled and running
Repeating the final host check...
docker (/bin/docker) is present
systemctl is present
lvcreate is present
Unit chrony.service is enabled and running
Host looks OK
Cluster fsid: 4099b12a-d104-11f0-aff5-75c1b810517d
Verifying IP 10.0.0.189 port 3300 ...
Verifying IP 10.0.0.189 port 6789 ...
Mon IP `10.0.0.189` is in CIDR network `10.0.0.0/24`
Mon IP `10.0.0.189` is in CIDR network `10.0.0.0/24`
Mon IP `10.0.0.189` is in CIDR network `10.0.0.253/32`
Mon IP `10.0.0.189` is in CIDR network `10.0.0.253/32`
Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network
Pulling container image quay.io/ceph/ceph:v17...
Ceph version: ceph version 17.2.8 (f817ceb7f187defb1d021d6328fa833eb8e943b3) quincy (stable)
Extracting ceph user uid/gid from container image...
Creating initial keys...
Creating initial monmap...
Creating mon...
Waiting for mon to start...
Waiting for mon...
mon is available
Assimilating anything we can from ceph.conf...
Generating new minimal ceph.conf...
Restarting the monitor...
Setting public_network to 10.0.0.0/24,10.0.0.253/32 in mon config section
Wrote config to /etc/ceph/ceph.conf
Wrote keyring to /etc/ceph/ceph.client.admin.keyring
Creating mgr...
Verifying port 0.0.0.0:9283 ...
Verifying port 0.0.0.0:8765 ...
Verifying port 0.0.0.0:8443 ...
Waiting for mgr to start...
Waiting for mgr...
mgr not available, waiting (1/15)...
mgr not available, waiting (2/15)...
mgr not available, waiting (3/15)...
mgr is available
Enabling cephadm module...
Waiting for the mgr to restart...
Waiting for mgr epoch 4...
mgr epoch 4 is available
Setting orchestrator backend to cephadm...
Generating ssh key...
Wrote public SSH key to /etc/ceph/ceph.pub
Adding key to root@localhost authorized_keys...
Adding host gc-test...
Deploying mon service with default placement...
Deploying mgr service with default placement...
Deploying crash service with default placement...
Deploying prometheus service with default placement...
Deploying grafana service with default placement...
Deploying node-exporter service with default placement...
Deploying alertmanager service with default placement...
Enabling the dashboard module...
Waiting for the mgr to restart...
Waiting for mgr epoch 8...
mgr epoch 8 is available
Generating a dashboard self-signed certificate...
Creating initial admin user...
Fetching dashboard port number...
Ceph Dashboard is now available at:
URL: https://gc-test:8443/
User: admin
Password: xb7g2tzrvj
Enabling client.admin keyring and conf on hosts with "admin" label
Saving cluster configuration to /var/lib/ceph/4099b12a-d104-11f0-aff5-75c1b810517d/config directory
Enabling autotune for osd_memory_target
You can access the Ceph CLI as following in case of multi-cluster or non-default config:
sudo /usr/local/bin/cephadm shell --fsid 4099b12a-d104-11f0-aff5-75c1b810517d -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring
Or, if you are only running a single cluster on this host:
sudo /usr/local/bin/cephadm shell
Please consider enabling telemetry to help improve Ceph:
ceph telemetry on
For more information see:
https://docs.ceph.com/docs/master/mgr/telemetry/
Bootstrap complete.上面这个命令运行完以后
MON（监视器）已创建并运行
MGR（管理器）已创建并运行
Dashboard（Web 管理界面）已启动
集群基础配置（ceph.conf、keyring）已生成
cephadm 编排器已经接管集群管理
Prometheus、Grafana、Node Exporter、AlertManager 已部署
CEPH 集群现在是可用的，但还缺 OSD（存储）
root@gc-test:/home/ceph# ceph -s
cluster:
id: 4099b12a-d104-11f0-aff5-75c1b810517d
health: HEALTH_WARN
Failed to place 1 daemon(s)
OSD count 0 < osd_pool_default_size 3
services:
mon: 1 daemons, quorum gc-test (age 50m)
mgr: gc-test.qxagkr(active, since 48m)
osd: 0 osds: 0 up, 0 in
data:
pools: 0 pools, 0 pgs
objects: 0 objects, 0 B
usage: 0 B used, 0 B / 0 B avail
pgs: 副本数 > OSD 数量 → 所以 Ceph 报 WARN,还是因为osd的问题，然后我们服务器上也没有空闲的磁盘
我们需要通过lsblk命令来判断当前服务器上是否有空闲的磁盘
空闲磁盘的定义是
Ceph OSD 要求磁盘必须：
- TYPE = disk（真实磁盘）
- 未挂载
- 无分区
- 无文件系统
- 空设备
我本地之前拉取的一个ceph镜像坏了，所以我就重新拉取了一个然后可以通过命令在处理bootstrap节点的时候指定镜像的
root@gc-test:/home# docker pull quay.io/ceph/ceph:v17.2.6
root@gc-test:/home# cephadm --image quay.io/ceph/ceph:v17.2.6 bootstrap --mon-ip 10.0.0.189接下来我们需要先额外申请一些空闲的磁盘，因为通过lsblk 查看可以知道sda这块磁盘已经用作系统盘，被使用了，所以得添加新的盘，OSD 磁盘建议用裸磁盘（无需提前分区 / 格式化），cephadm 会自动处理
niuniu@niuniu-virtual-machine:~/Desktop/ceph$ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
loop0 7:0 0 4K 1 loop /snap/bare/5
loop1 7:1 0 346.3M 1 loop /snap/gnome-3-38-2004/119
loop2 7:2 0 63.8M 1 loop /snap/core20/2682
loop3 7:3 0 73.9M 1 loop /snap/core22/1963
loop4 7:4 0 63.8M 1 loop /snap/core20/2582
loop5 7:5 0 74M 1 loop /snap/core22/2163
loop6 7:6 0 505.1M 1 loop /snap/gnome-42-2204/176
loop7 7:7 0 349.7M 1 loop /snap/gnome-3-38-2004/143
loop8 7:8 0 12.9M 1 loop /snap/snap-store/1113
loop9 7:9 0 12.2M 1 loop /snap/snap-store/1216
loop10 7:10 0 516M 1 loop /snap/gnome-42-2204/202
loop11 7:11 0 91.7M 1 loop /snap/gtk-common-themes/1535
loop12 7:12 0 50.9M 1 loop /snap/snapd/24505
loop13 7:13 0 50.9M 1 loop /snap/snapd/25577
sda 8:0 0 150G 0 disk
├─sda1 8:1 0 512M 0 part /boot/efi
├─sda2 8:2 0 1K 0 part
└─sda5 8:5 0 149.5G 0 part /
sdb 8:16 0 20G 0 disk
sr0 11:0 1 4.1G 0 rom /media/niuniu/Ubuntu 20.04.6 LTS amd64然后我们也可以查看当前ceph有哪些盘可以使用
niuniu@niuniu-virtual-machine:~/Desktop/ceph$ sudo ceph orch device ls
HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS
niuniu-virtual-machine /dev/sdb hdd 20.0G Yes 23m ago // todo：将其他节点添加到集群里面，需要通过ssh免密，不过我这个是虚拟机，目前就没有其他节点，添加的命令先贴出来
# 在 bootstrap 节点（192.168.244.128）添加其他节点
sudo ceph orch host add node2 192.168.244.129
sudo ceph orch host add node3 192.168.244.130添加osd，之前我们检查过有空闲的磁盘可以用作osd，下面我们通过命令进行添加
sudo ceph orch daemon add osd niuniu-virtual-machine:/dev/sdb这里的niuniu-virtual-machine 是hostname，可以在命令行通过hostname的方式进行获得，
niuniu@niuniu-virtual-machine:~/Desktop/ceph$ sudo ceph -s
cluster:
id: 38d01ece-d286-11f0-8ad6-a312283fe45c
health: HEALTH_WARN
Reduced data availability: 1 pg inactive
Degraded data redundancy: 1 pg undersized
services:
mon: 1 daemons, quorum niuniu-virtual-machine (age 20m)
mgr: niuniu-virtual-machine.igrcze(active, since 20m)
osd: 3 osds: 3 up (since 10m), 3 in (since 10m)
data:
pools: 1 pools, 1 pgs
objects: 0 objects, 0 B
usage: 871 MiB used, 59 GiB / 60 GiB avail
pgs: 100.000% pgs not active
1 undersized+peered部署3个osd以后可以看到ceph -s可以显示了
pg数量设置太少了，我们可以设置pg数量
niuniu@niuniu-virtual-machine:~/Desktop/ceph$ sudo ceph osd pool ls
.mgr
niuniu@niuniu-virtual-machine:~/Desktop/ceph$ sudo ceph osd pool get .mgr size
size: 3
niuniu@niuniu-virtual-machine:~/Desktop/ceph$ sudo ceph osd pool set .mgr pg_num 32
set pool 1 pg_num to 32
niuniu@niuniu-virtual-machine:~/Desktop/ceph$ sudo ceph -s
cluster:
id: 38d01ece-d286-11f0-8ad6-a312283fe45c
health: HEALTH_WARN
Reduced data availability: 32 pgs inactive
services:
mon: 1 daemons, quorum niuniu-virtual-machine (age 47m)
mgr: niuniu-virtual-machine.igrcze(active, since 47m)
osd: 3 osds: 3 up (since 37m), 3 in (since 38m)
data:
pools: 1 pools, 32 pgs
objects: 0 objects, 0 B
usage: 871 MiB used, 59 GiB / 60 GiB avail
pgs: 100.000% pgs not active
32 undersized+peered
set pool 1 pg_num to 32
niuniu@niuniu-virtual-machine:~/Desktop/ceph$ sudo ceph -s
cluster:
id: 38d01ece-d286-11f0-8ad6-a312283fe45c
health: HEALTH_WARN
Reduced data availability: 32 pgs inactive
services:
mon: 1 daemons, quorum niuniu-virtual-machine (age 47m)
mgr: niuniu-virtual-machine.igrcze(active, since 47m)
osd: 3 osds: 3 up (since 37m), 3 in (since 38m)
data:
pools: 1 pools, 32 pgs
objects: 0 objects, 0 B
usage: 871 MiB used, 59 GiB / 60 GiB avail
pgs: 100.000% pgs not active
32 undersized+peered
niuniu@niuniu-virtual-machine:~/Desktop/ceph$ sudo ceph osd pool set .mgr size 1
Error EPERM: configuring pool size as 1 is disabled by default.
niuniu@niuniu-virtual-machine:~/Desktop/ceph$ sudo ceph osd pool set .mgr size 2
set pool 1 size to 2
niuniu@niuniu-virtual-machine:~/Desktop/ceph$ sudo ceph -s
cluster:
id: 38d01ece-d286-11f0-8ad6-a312283fe45c
health: HEALTH_WARN
Degraded data redundancy: 2/4 objects degraded (50.000%), 2 pgs degraded
services:
mon: 1 daemons, quorum niuniu-virtual-machine (age 64m)
mgr: niuniu-virtual-machine.igrcze(active, since 64m)
osd: 3 osds: 3 up (since 54m), 3 in (since 54m)
data:
pools: 1 pools, 32 pgs
objects: 2 objects, 320 KiB
usage: 872 MiB used, 59 GiB / 60 GiB avail
pgs: 2/4 objects degraded (50.000%)
30 active+undersized
2 active+undersized+degraded
niuniu@niuniu-virtual-machine:~/Desktop/ceph$ sudo ceph osd pool set .mgr size 3
set pool 1 size to 3
niuniu@niuniu-virtual-machine:~/Desktop/ceph$ sudo ceph -s
cluster:
id: 38d01ece-d286-11f0-8ad6-a312283fe45c
health: HEALTH_WARN
Degraded data redundancy: 2/4 objects degraded (50.000%), 2 pgs degraded, 32 pgs undersized
services:
mon: 1 daemons, quorum niuniu-virtual-machine (age 67m)
mgr: niuniu-virtual-machine.igrcze(active, since 67m)
osd: 3 osds: 3 up (since 57m), 3 in (since 57m)
data:
pools: 1 pools, 32 pgs
objects: 2 objects, 449 KiB
usage: 872 MiB used, 59 GiB / 60 GiB avail
pgs: 2/4 objects degraded (50.000%)
30 active+undersized
2 active+undersized+degraded
niuniu@niuniu-virtual-machine:~/Desktop/ceph$ sudo ceph -s
cluster:
id: 38d01ece-d286-11f0-8ad6-a312283fe45c
health: HEALTH_WARN
Reduced data availability: 32 pgs inactive
Degraded data redundancy: 4/6 objects degraded (66.667%), 2 pgs degraded, 32 pgs undersized
services:
mon: 1 daemons, quorum niuniu-virtual-machine (age 68m)
mgr: niuniu-virtual-machine.igrcze(active, since 68m)
osd: 3 osds: 3 up (since 58m), 3 in (since 58m)
data:
pools: 1 pools, 32 pgs
objects: 2 objects, 449 KiB
usage: 872 MiB used, 59 GiB / 60 GiB avail
pgs: 100.000% pgs not active
4/6 objects degraded (66.667%)
30 undersized+peered
2 undersized+degraded+peered
niuniu@niuniu-virtual-machine:~/Desktop/ceph$ sudo ceph osd pool set .mgr size 1
Error EPERM: configuring pool size as 1 is disabled by default.
niuniu@niuniu-virtual-machine:~/Desktop/ceph$ sudo ceph osd pool set .mgr size 3
set pool 1 size to 3
niuniu@niuniu-virtual-machine:~/Desktop/ceph$ sudo ceph -s
cluster:
id: 38d01ece-d286-11f0-8ad6-a312283fe45c
health: HEALTH_WARN
Reduced data availability: 32 pgs inactive
Degraded data redundancy: 4/6 objects degraded (66.667%), 2 pgs degraded, 32 pgs undersized
services:
mon: 1 daemons, quorum niuniu-virtual-machine (age 72m)
mgr: niuniu-virtual-machine.igrcze(active, since 72m)
osd: 3 osds: 3 up (since 62m), 3 in (since 63m)
data:
pools: 1 pools, 32 pgs
objects: 2 objects, 449 KiB
usage: 872 MiB used, 59 GiB / 60 GiB avail
pgs: 100.000% pgs not active
4/6 objects degraded (66.667%)
30 undersized+peered
2 undersized+degraded+peered
niuniu@niuniu-virtual-machine:~/Desktop/ceph$ sudo ceph pg repair all
Invalid command: pgid has no .
pg repair <pgid> : start repair on <pgid>
Error EINVAL: invalid command
niuniu@niuniu-virtual-machine:~/Desktop/ceph$ sudo ceph -s
cluster:
id: 38d01ece-d286-11f0-8ad6-a312283fe45c
health: HEALTH_WARN
Reduced data availability: 32 pgs inactive
Degraded data redundancy: 4/6 objects degraded (66.667%), 2 pgs degraded, 32 pgs undersized
services:
mon: 1 daemons, quorum niuniu-virtual-machine (age 76m)
mgr: niuniu-virtual-machine.igrcze(active, since 76m)
osd: 3 osds: 3 up (since 66m), 3 in (since 66m)
data:
pools: 1 pools, 32 pgs
objects: 2 objects, 449 KiB
usage: 872 MiB used, 59 GiB / 60 GiB avail
pgs: 100.000% pgs not active
4/6 objects degraded (66.667%)
30 undersized+peered
2 undersized+degraded+peered
root@niuniu-virtual-machine:~# wipefs -a /dev/sdb
root@niuniu-virtual-machine:~# wipefs -a /dev/sdc
root@niuniu-virtual-machine:~# wipefs -a /dev/sdd
root@niuniu-virtual-machine:~# sgdisk --zap-all /dev/sdb
Creating new GPT entries in memory.
GPT data structures destroyed! You may now partition the disk using fdisk or
other utilities.
root@niuniu-virtual-machine:~# sgdisk --zap-all /dev/sdc
Creating new GPT entries in memory.
GPT data structures destroyed! You may now partition the disk using fdisk or
other utilities.
root@niuniu-virtual-machine:~# sgdisk --zap-all /dev/sdd
Creating new GPT entries in memory.
GPT data structures destroyed! You may now partition the disk using fdisk or
other utilities.
root@niuniu-virtual-machine:~# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
loop0 7:0 0 4K 1 loop /snap/bare/5
loop1 7:1 0 63.8M 1 loop /snap/core20/2582
loop2 7:2 0 63.8M 1 loop /snap/core20/2682
loop3 7:3 0 73.9M 1 loop /snap/core22/1963
loop4 7:4 0 346.3M 1 loop /snap/gnome-3-38-2004/119
loop5 7:5 0 74M 1 loop /snap/core22/2163
loop6 7:6 0 516M 1 loop /snap/gnome-42-2204/202
loop7 7:7 0 349.7M 1 loop /snap/gnome-3-38-2004/143
loop8 7:8 0 12.2M 1 loop /snap/snap-store/1216
loop9 7:9 0 516.2M 1 loop /snap/gnome-42-2204/226
loop10 7:10 0 91.7M 1 loop /snap/gtk-common-themes/1535
loop11 7:11 0 50.9M 1 loop /snap/snapd/24505
loop12 7:12 0 50.9M 1 loop /snap/snapd/25577
loop13 7:13 0 12.9M 1 loop /snap/snap-store/1113
sda 8:0 0 150G 0 disk
├─sda1 8:1 0 512M 0 part /boot/efi
├─sda2 8:2 0 1K 0 part
└─sda5 8:5 0 149.5G 0 part /
sdb 8:16 0 20G 0 disk
sdc 8:32 0 20G 0 disk
sdd 8:48 0 20G 0 disk
sr0 11:0 1 4.1G 0 rom /media/niuniu/Ubuntu 20.04.6 LTS amd64