bclinux aarch64 ceph 14.2.10 对象存储 http网关 CEPH OBJECT GATEWAY Civetweb

相关内容

bclinux aarch64 ceph 14.2.10 文件存储 Ceph File System, 需要部署mds: ceph-deploy mds-CSDN博客

ceph-deploy bclinux aarch64 ceph 14.2.10【3】vdbench fsd 文件系统测试-CSDN博客

ceph-deploy bclinux aarch64 ceph 14.2.10【2】vdbench rbd 块设备rbd 测试失败-CSDN博客

ceph-deploy bclinux aarch64 ceph 14.2.10-CSDN博客

部署对象存储网关

复制代码
cd /etc/ceph
ceph-deploy rgw create ceph-0 ceph-1 ceph-2 ceph-3

成功日志

复制代码
[root@ceph-0 ceph]# ceph-deploy rgw create ceph-0 ceph-1 ceph-2 ceph-3
[ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf
[ceph_deploy.cli][INFO  ] Invoked (2.0.1): /usr/bin/ceph-deploy rgw create ceph-0 ceph-1 ceph-2 ceph-3
[ceph_deploy.cli][INFO  ] ceph-deploy options:
[ceph_deploy.cli][INFO  ]  username                      : None
[ceph_deploy.cli][INFO  ]  verbose                       : False
[ceph_deploy.cli][INFO  ]  rgw                           : [('ceph-0', 'rgw.ceph-0'), ('ceph-1', 'rgw.ceph-1'), ('ceph-2', 'rgw.ceph-2'), ('ceph-3', 'rgw.ceph-3')]
[ceph_deploy.cli][INFO  ]  overwrite_conf                : False
[ceph_deploy.cli][INFO  ]  subcommand                    : create
[ceph_deploy.cli][INFO  ]  quiet                         : False
[ceph_deploy.cli][INFO  ]  cd_conf                       : <ceph_deploy.conf.cephdeploy.Conf instance at 0xffffa1b8faa0>
[ceph_deploy.cli][INFO  ]  cluster                       : ceph
[ceph_deploy.cli][INFO  ]  func                          : <function rgw at 0xffffa1d554d0>
[ceph_deploy.cli][INFO  ]  ceph_conf                     : None
[ceph_deploy.cli][INFO  ]  default_release               : False
[ceph_deploy.rgw][DEBUG ] Deploying rgw, cluster ceph hosts ceph-0:rgw.ceph-0 ceph-1:rgw.ceph-1 ceph-2:rgw.ceph-2 ceph-3:rgw.ceph-3
[ceph-0][DEBUG ] connected to host: ceph-0 
[ceph-0][DEBUG ] detect platform information from remote host
21.10U3 LTS
bclinux
[ceph-0][DEBUG ] detect machine type
[ceph_deploy.rgw][INFO  ] Distro info: bclinux 21.10U3 21.10U3 LTS
[ceph_deploy.rgw][DEBUG ] remote host will use systemd
[ceph_deploy.rgw][DEBUG ] deploying rgw bootstrap to ceph-0
[ceph-0][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph-0][WARNIN] rgw keyring does not exist yet, creating one
[ceph-0][DEBUG ] create a keyring file
[ceph-0][DEBUG ] create path recursively if it doesn't exist
[ceph-0][INFO  ] Running command: ceph --cluster ceph --name client.bootstrap-rgw --keyring /var/lib/ceph/bootstrap-rgw/ceph.keyring auth get-or-create client.rgw.ceph-0 osd allow rwx mon allow rw -o /var/lib/ceph/radosgw/ceph-rgw.ceph-0/keyring
[ceph-0][INFO  ] Running command: systemctl enable [email protected]
[ceph-0][WARNIN] Created symlink /etc/systemd/system/ceph-radosgw.target.wants/[email protected] → /usr/lib/systemd/system/[email protected].
[ceph-0][INFO  ] Running command: systemctl start [email protected]
[ceph-0][INFO  ] Running command: systemctl enable ceph.target
[ceph_deploy.rgw][INFO  ] The Ceph Object Gateway (RGW) is now running on host ceph-0 and default port 7480
dhclient(1787) is already running - exiting. 

This version of ISC DHCP is based on the release available
on ftp.isc.org. Features have been added and other changes
have been made to the base software release in order to make
it work better with this distribution.

Please report issues with this software via: 
https://gitee.com/src-openeuler/dhcp/issues

exiting.
dhclient(1787) is already running - exiting. 

This version of ISC DHCP is based on the release available
on ftp.isc.org. Features have been added and other changes
have been made to the base software release in order to make
it work better with this distribution.

Please report issues with this software via: 
https://gitee.com/src-openeuler/dhcp/issues

exiting.
[ceph-1][DEBUG ] connected to host: ceph-1 
[ceph-1][DEBUG ] detect platform information from remote host
21.10U3 LTS
bclinux
[ceph-1][DEBUG ] detect machine type
[ceph_deploy.rgw][INFO  ] Distro info: bclinux 21.10U3 21.10U3 LTS
[ceph_deploy.rgw][DEBUG ] remote host will use systemd
[ceph_deploy.rgw][DEBUG ] deploying rgw bootstrap to ceph-1
[ceph-1][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph-1][WARNIN] rgw keyring does not exist yet, creating one
[ceph-1][DEBUG ] create a keyring file
[ceph-1][DEBUG ] create path recursively if it doesn't exist
[ceph-1][INFO  ] Running command: ceph --cluster ceph --name client.bootstrap-rgw --keyring /var/lib/ceph/bootstrap-rgw/ceph.keyring auth get-or-create client.rgw.ceph-1 osd allow rwx mon allow rw -o /var/lib/ceph/radosgw/ceph-rgw.ceph-1/keyring
[ceph-1][INFO  ] Running command: systemctl enable [email protected]
[ceph-1][WARNIN] Created symlink /etc/systemd/system/ceph-radosgw.target.wants/[email protected] → /usr/lib/systemd/system/[email protected].
[ceph-1][INFO  ] Running command: systemctl start [email protected]
[ceph-1][INFO  ] Running command: systemctl enable ceph.target
[ceph_deploy.rgw][INFO  ] The Ceph Object Gateway (RGW) is now running on host ceph-1 and default port 7480
dhclient(1742) is already running - exiting. 

This version of ISC DHCP is based on the release available
on ftp.isc.org. Features have been added and other changes
have been made to the base software release in order to make
it work better with this distribution.

Please report issues with this software via: 
https://gitee.com/src-openeuler/dhcp/issues

exiting.
dhclient(1742) is already running - exiting. 

This version of ISC DHCP is based on the release available
on ftp.isc.org. Features have been added and other changes
have been made to the base software release in order to make
it work better with this distribution.

Please report issues with this software via: 
https://gitee.com/src-openeuler/dhcp/issues

exiting.
[ceph-2][DEBUG ] connected to host: ceph-2 
[ceph-2][DEBUG ] detect platform information from remote host
21.10U3 LTS
bclinux
[ceph-2][DEBUG ] detect machine type
[ceph_deploy.rgw][INFO  ] Distro info: bclinux 21.10U3 21.10U3 LTS
[ceph_deploy.rgw][DEBUG ] remote host will use systemd
[ceph_deploy.rgw][DEBUG ] deploying rgw bootstrap to ceph-2
[ceph-2][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph-2][WARNIN] rgw keyring does not exist yet, creating one
[ceph-2][DEBUG ] create a keyring file
[ceph-2][DEBUG ] create path recursively if it doesn't exist
[ceph-2][INFO  ] Running command: ceph --cluster ceph --name client.bootstrap-rgw --keyring /var/lib/ceph/bootstrap-rgw/ceph.keyring auth get-or-create client.rgw.ceph-2 osd allow rwx mon allow rw -o /var/lib/ceph/radosgw/ceph-rgw.ceph-2/keyring
[ceph-2][INFO  ] Running command: systemctl enable [email protected]
[ceph-2][WARNIN] Created symlink /etc/systemd/system/ceph-radosgw.target.wants/[email protected] → /usr/lib/systemd/system/[email protected].
[ceph-2][INFO  ] Running command: systemctl start [email protected]
[ceph-2][INFO  ] Running command: systemctl enable ceph.target
[ceph_deploy.rgw][INFO  ] The Ceph Object Gateway (RGW) is now running on host ceph-2 and default port 7480
dhclient(1722) is already running - exiting. 

This version of ISC DHCP is based on the release available
on ftp.isc.org. Features have been added and other changes
have been made to the base software release in order to make
it work better with this distribution.

Please report issues with this software via: 
https://gitee.com/src-openeuler/dhcp/issues

exiting.
dhclient(1722) is already running - exiting. 

This version of ISC DHCP is based on the release available
on ftp.isc.org. Features have been added and other changes
have been made to the base software release in order to make
it work better with this distribution.

Please report issues with this software via: 
https://gitee.com/src-openeuler/dhcp/issues

exiting.
[ceph-3][DEBUG ] connected to host: ceph-3 
[ceph-3][DEBUG ] detect platform information from remote host
21.10U3 LTS
bclinux
[ceph-3][DEBUG ] detect machine type
[ceph_deploy.rgw][INFO  ] Distro info: bclinux 21.10U3 21.10U3 LTS
[ceph_deploy.rgw][DEBUG ] remote host will use systemd
[ceph_deploy.rgw][DEBUG ] deploying rgw bootstrap to ceph-3
[ceph-3][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph-3][WARNIN] rgw keyring does not exist yet, creating one
[ceph-3][DEBUG ] create a keyring file
[ceph-3][DEBUG ] create path recursively if it doesn't exist
[ceph-3][INFO  ] Running command: ceph --cluster ceph --name client.bootstrap-rgw --keyring /var/lib/ceph/bootstrap-rgw/ceph.keyring auth get-or-create client.rgw.ceph-3 osd allow rwx mon allow rw -o /var/lib/ceph/radosgw/ceph-rgw.ceph-3/keyring
[ceph-3][INFO  ] Running command: systemctl enable [email protected]
[ceph-3][WARNIN] Created symlink /etc/systemd/system/ceph-radosgw.target.wants/[email protected] → /usr/lib/systemd/system/[email protected].
[ceph-3][INFO  ] Running command: systemctl start [email protected]
[ceph-3][INFO  ] Running command: systemctl enable ceph.target
[ceph_deploy.rgw][INFO  ] The Ceph Object Gateway (RGW) is now running on host ceph-3 and default port 7480
[root@ceph-0 ceph]# cp: cannot stat '/tmp/security_updates': No such file or directory
cp: cannot stat '/tmp/security_updates': No such file or directory
cp: cannot stat '/tmp/security_updates': No such file or directory

创建成功

验证服务状态(失败)

日志目录

/var/log/ceph

日志文件

ceph-client.rgw.ceph-0.log

清空日志文件,重启服务

复制代码
cd /var/log/ceph
echo "" > ceph-client.rgw.ceph-0.log
systemctl start ceph-radosgw@rgw-ceph-0

先看服务日志

复制代码
journalctl -xeu ceph-radosgw@rgw-ceph-0

日志

Nov 14 13:38:08 ceph-0 radosgw[58375]: 2023-11-14 13:38:08.450 ffff7ab37010 -1 auth: unable to find a keyring on /var/lib/ceph/radosgw/ceph-rgw-ceph-0/keyring: (2) No such file or directory

Nov 14 13:38:08 ceph-0 radosgw[58375]: 2023-11-14 13:38:08.450 ffff7ab37010 -1 AuthRegistry(0xffff7a330148) no keyring found at /var/lib/ceph/radosgw/ceph-rgw-ceph-0/keyring, disabling cephx

Nov 14 13:38:08 ceph-0 radosgw[58375]: 2023-11-14 13:38:08.450 ffff7ab37010 -1 auth: unable to find a keyring on /var/lib/ceph/radosgw/ceph-rgw-ceph-0/keyring: (2) No such file or directory

Nov 14 13:38:08 ceph-0 radosgw[58375]: 2023-11-14 13:38:08.450 ffff7ab37010 -1 AuthRegistry(0xffffdc7f98c8) no keyring found at /var/lib/ceph/radosgw/ceph-rgw-ceph-0/keyring, disabling cephx

Nov 14 13:38:08 ceph-0 radosgw[58375]: failed to fetch mon config (--no-mon-config to skip)

Nov 14 13:38:08 ceph-0 systemd[1]: [email protected]: Main process exited, code=exited, status=1/FAILURE

没有按照官方文档执行网关安装有关?ceph-deploy install --rgw node..

但是这个版本的ceph-deploy指定的ceph版本太低,且是el6

解决 /var/lib/ceph/radosgw/ceph-rgw-ceph-0/keyring: (2) No such file or directory

查看/etc/ceph目录下的keyring文件,发现已生成了ceph.bootstrap-rgw.keyring

错误的路径/var/lib/ceph/radosgw/ceph-rgw-ceph-0/keyring

正确的路径/var/lib/ceph/radosgw/ceph-rgw.ceph-0/keyring

不知道为什么路径有一个符号错误

做个软连接

复制代码
cd /var/lib/ceph/radosgw
ln -s ceph-rgw.ceph-0 ceph-rgw-ceph-0

错误:

Nov 14 14:11:43 ceph-0 radosgw[59264]: 2023-11-14 14:11:43.076 ffff99055730 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2]

Nov 14 14:11:43 ceph-0 radosgw[59264]: 2023-11-14 14:11:43.080 ffff99856730 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2]

Nov 14 14:11:43 ceph-0 radosgw[59264]: failed to fetch mon config (--no-mon-config to skip)

Nov 14 14:11:43 ceph-0 systemd[1]: [email protected]: Main process exited, code=exited, status=1/FAILURE

重新执行部署

复制代码
cd /etc/ceph
ceph-deploy rgw create ceph-0 ceph-1 ceph-2 ceph-3

重新部署后,会生成/var/log/ceph/ceph-client.rgw.ceph-0.log日志文件

其中有日志

Numerical result out of range

2023-11-14 14:32:24.461 ffff8abe8010 0 deferred set uid:gid to 1000:1000 (ceph:ceph)

2023-11-14 14:32:24.461 ffff8abe8010 0 ceph version 14.2.10 (b340acf629a010a74d90da5782a2c5fe0b54ac20) nautilus (stable), process radosgw, pid 60035

2023-11-14 14:32:25.509 ffff8abe8010 0 rgw_init_ioctx ERROR: librados::Rados::pool_create returned (34) Numerical result out of range (this can be due to a pool or placement group misconfiguration, e.g. pg_num < pgp_num or mon_max_pg_per_osd exceeded)

2023-11-14 14:32:25.509 ffff8abe8010 0 failed reading realm info: ret -34 (34) Numerical result out of range

2023-11-14 14:32:25.509 ffff8abe8010 0 ERROR: failed to start notify service ((34) Numerical result out of range

2023-11-14 14:32:25.509 ffff8abe8010 0 ERROR: failed to init services (ret=(34) Numerical result out of range)

2023-11-14 14:32:25.513 ffff8abe8010 -1 Couldn't init storage provider (RADOS)

设置默认pg_num pgs_num

/etc/ceph/ceph.conf

复制代码
osd pool default size = 3
osd pool default min_size = 1
osd pool default pg num = 128
osd pool default pgp num = 128

重新部署(--overwrite-conf)

复制代码
cd /etc/ceph
ceph-deploy --overwrite-conf rgw create ceph-0 ceph-1 ceph-2 ceph-3

故障依旧

成功的方案

删掉所有pool,重新部署,参考以下历史记录

652 ceph osd pool ls

653 ceph osd pool rm vdbench

654 ceph osd pool rm vdbench vdbench --yes-i-really-really-mean-it

655 ls /dev/

656 mount

657 df -h

658 umount /mnt/rbd0

659 ceph osd ls

660 ceph osd pool ls

661 ceph osd volume ls

662 ceph osd dump

663 ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it

664 ceph fs ls

665 ceph fs rm cephfs

666 ceph fs fail

667 ceph fs fail cephfs

668 ceph -s

669 ceph fs status cephfs

670 ceph fs ls

671 ceph fs rm cephfs

672 ceph fs rm cephfs --yes-i-really-mean-it

673 ceph fs ls

674 ceph osd pool ls

675 ceph osd pool rm cephfs_data cephfs_data --yes-i-really-mean-it

676 ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it

677 ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it

678 ceph -

679 ceph -s

680 cd /etc/ceph

681 ceph-deploy rgw create ceph-0

682 cat /var/log/ceph/ceph-client.rgw.ceph-0.log

683 systemctl status [email protected]

再来一次集群环境的部署

复制代码
cd /etc/ceph
ceph-deploy --overwrite-conf rgw create ceph-0 ceph-1 ceph-2 ceph-3

部分日志

ceph -s rgw已经起来了

======================

以下为参考官方文档的失败经理,留档参考

部署对象存储网关rgw(失败)

复制代码
ceph-deploy install --rgw ceph-0 ceph-1 ceph-2 ceph-3

故障 Unable to find a match: epel-release

复制代码
dnf install https://dl.fedoraproject.org/pub/epel/epel-release-latest-8.noarch.rpm -y

故障Status code: 404

坑爹的阿里云镜像,没有epel 8,bclinux又是el8版本!

采用huawei epel镜像

复制代码
[epel-everything]
name=huawei epel mirror
baseurl=https://repo.huaweicloud.com/epel/8/Everything/aarch64/
enabled=1
gpgcheck=0

yum install epel-release --nobest --skip-broken

ceph-deploy 会强制执行yum install epel-release(失败)

注释掉epel-release的安装

修改/usr/lib/python2.7/site-packages/ceph_deploy/hosts/centos/install.py 60行,注释掉epel-release的安装(前面已经手动--skip-broken安装好了)

生效了

故障 Unable to find a match: yum-plugin-priorities

参考上面思路,直接注释掉/usr/lib/python2.7/site-packages/ceph_deploy/hosts/centos/install.py 61,62行

自动找到老版本ceph-release-1-0.el6 Status code: 404

手动安装

修改/usr/lib/python2.7/site-packages/ceph_deploy/install.py

安装版本参数args.release = 'mimic' 改为 args.release = 'nautilus

关于python3版本

bclinux python3.7.9

cenos7 python3.6.8

openeuler 20.03 lts sp3 python 3.7.9

参考

Ceph Object Gateway --- Ceph Documentation

如何在 CentOS 8 和 RHEL 8 服务器上启用 EPEL 仓库 - 知乎 (zhihu.com)

ceph-deploy部署ceph-nautilus_ceph nautilus-CSDN博客

相关推荐
斯普信专业组2 天前
Ceph与Bacula运维实战:数据迁移与备份配置优化指南
运维·ceph·github
A叶子叶2 天前
Kong网关部署研究
python·spring cloud·微服务·gateway·kong
甜可儿4 天前
Gateway实战入门(四)、断言-请求头以及请求权重分流等
java·spring cloud·gateway
INFINI Labs4 天前
实现极限网关(INFINI Gateway)配置动态加载
gateway
brhhh_sehe9 天前
【Java面试系列】初识GateWay网关
java·面试·gateway
程序媛学姐9 天前
SpringCloud网关:Gateway路由配置与过滤器链
java·spring cloud·gateway
野猪佩挤10 天前
Ceph集群2025(Squid版)导出高可用NFS集群(上集)
java·服务器·ceph
时雨h11 天前
微服务架构-网关学习 以Spring Cloud Gateway为例 详细功能模块解读
微服务·架构·gateway
ronshi14 天前
Spring Cloud Gateway 使用ribbon以及nacos实现灰度发布
ribbon·nacos·gateway·灰度
carfied-feifei15 天前
云安全相关博客阅读(四)
云原生·gateway·云安全