bclinux aarch64 ceph 14.2.10 对象存储 http网关 CEPH OBJECT GATEWAY Civetweb

相关内容

bclinux aarch64 ceph 14.2.10 文件存储 Ceph File System, 需要部署mds: ceph-deploy mds-CSDN博客

ceph-deploy bclinux aarch64 ceph 14.2.10【3】vdbench fsd 文件系统测试-CSDN博客

ceph-deploy bclinux aarch64 ceph 14.2.10【2】vdbench rbd 块设备rbd 测试失败-CSDN博客

ceph-deploy bclinux aarch64 ceph 14.2.10-CSDN博客

部署对象存储网关

cd /etc/ceph
ceph-deploy rgw create ceph-0 ceph-1 ceph-2 ceph-3

成功日志

[root@ceph-0 ceph]# ceph-deploy rgw create ceph-0 ceph-1 ceph-2 ceph-3
[ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf
[ceph_deploy.cli][INFO  ] Invoked (2.0.1): /usr/bin/ceph-deploy rgw create ceph-0 ceph-1 ceph-2 ceph-3
[ceph_deploy.cli][INFO  ] ceph-deploy options:
[ceph_deploy.cli][INFO  ]  username                      : None
[ceph_deploy.cli][INFO  ]  verbose                       : False
[ceph_deploy.cli][INFO  ]  rgw                           : [('ceph-0', 'rgw.ceph-0'), ('ceph-1', 'rgw.ceph-1'), ('ceph-2', 'rgw.ceph-2'), ('ceph-3', 'rgw.ceph-3')]
[ceph_deploy.cli][INFO  ]  overwrite_conf                : False
[ceph_deploy.cli][INFO  ]  subcommand                    : create
[ceph_deploy.cli][INFO  ]  quiet                         : False
[ceph_deploy.cli][INFO  ]  cd_conf                       : <ceph_deploy.conf.cephdeploy.Conf instance at 0xffffa1b8faa0>
[ceph_deploy.cli][INFO  ]  cluster                       : ceph
[ceph_deploy.cli][INFO  ]  func                          : <function rgw at 0xffffa1d554d0>
[ceph_deploy.cli][INFO  ]  ceph_conf                     : None
[ceph_deploy.cli][INFO  ]  default_release               : False
[ceph_deploy.rgw][DEBUG ] Deploying rgw, cluster ceph hosts ceph-0:rgw.ceph-0 ceph-1:rgw.ceph-1 ceph-2:rgw.ceph-2 ceph-3:rgw.ceph-3
[ceph-0][DEBUG ] connected to host: ceph-0 
[ceph-0][DEBUG ] detect platform information from remote host
21.10U3 LTS
bclinux
[ceph-0][DEBUG ] detect machine type
[ceph_deploy.rgw][INFO  ] Distro info: bclinux 21.10U3 21.10U3 LTS
[ceph_deploy.rgw][DEBUG ] remote host will use systemd
[ceph_deploy.rgw][DEBUG ] deploying rgw bootstrap to ceph-0
[ceph-0][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph-0][WARNIN] rgw keyring does not exist yet, creating one
[ceph-0][DEBUG ] create a keyring file
[ceph-0][DEBUG ] create path recursively if it doesn't exist
[ceph-0][INFO  ] Running command: ceph --cluster ceph --name client.bootstrap-rgw --keyring /var/lib/ceph/bootstrap-rgw/ceph.keyring auth get-or-create client.rgw.ceph-0 osd allow rwx mon allow rw -o /var/lib/ceph/radosgw/ceph-rgw.ceph-0/keyring
[ceph-0][INFO  ] Running command: systemctl enable ceph-radosgw@rgw.ceph-0
[ceph-0][WARNIN] Created symlink /etc/systemd/system/ceph-radosgw.target.wants/ceph-radosgw@rgw.ceph-0.service → /usr/lib/systemd/system/ceph-radosgw@.service.
[ceph-0][INFO  ] Running command: systemctl start ceph-radosgw@rgw.ceph-0
[ceph-0][INFO  ] Running command: systemctl enable ceph.target
[ceph_deploy.rgw][INFO  ] The Ceph Object Gateway (RGW) is now running on host ceph-0 and default port 7480
dhclient(1787) is already running - exiting. 

This version of ISC DHCP is based on the release available
on ftp.isc.org. Features have been added and other changes
have been made to the base software release in order to make
it work better with this distribution.

Please report issues with this software via: 
https://gitee.com/src-openeuler/dhcp/issues

exiting.
dhclient(1787) is already running - exiting. 

This version of ISC DHCP is based on the release available
on ftp.isc.org. Features have been added and other changes
have been made to the base software release in order to make
it work better with this distribution.

Please report issues with this software via: 
https://gitee.com/src-openeuler/dhcp/issues

exiting.
[ceph-1][DEBUG ] connected to host: ceph-1 
[ceph-1][DEBUG ] detect platform information from remote host
21.10U3 LTS
bclinux
[ceph-1][DEBUG ] detect machine type
[ceph_deploy.rgw][INFO  ] Distro info: bclinux 21.10U3 21.10U3 LTS
[ceph_deploy.rgw][DEBUG ] remote host will use systemd
[ceph_deploy.rgw][DEBUG ] deploying rgw bootstrap to ceph-1
[ceph-1][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph-1][WARNIN] rgw keyring does not exist yet, creating one
[ceph-1][DEBUG ] create a keyring file
[ceph-1][DEBUG ] create path recursively if it doesn't exist
[ceph-1][INFO  ] Running command: ceph --cluster ceph --name client.bootstrap-rgw --keyring /var/lib/ceph/bootstrap-rgw/ceph.keyring auth get-or-create client.rgw.ceph-1 osd allow rwx mon allow rw -o /var/lib/ceph/radosgw/ceph-rgw.ceph-1/keyring
[ceph-1][INFO  ] Running command: systemctl enable ceph-radosgw@rgw.ceph-1
[ceph-1][WARNIN] Created symlink /etc/systemd/system/ceph-radosgw.target.wants/ceph-radosgw@rgw.ceph-1.service → /usr/lib/systemd/system/ceph-radosgw@.service.
[ceph-1][INFO  ] Running command: systemctl start ceph-radosgw@rgw.ceph-1
[ceph-1][INFO  ] Running command: systemctl enable ceph.target
[ceph_deploy.rgw][INFO  ] The Ceph Object Gateway (RGW) is now running on host ceph-1 and default port 7480
dhclient(1742) is already running - exiting. 

This version of ISC DHCP is based on the release available
on ftp.isc.org. Features have been added and other changes
have been made to the base software release in order to make
it work better with this distribution.

Please report issues with this software via: 
https://gitee.com/src-openeuler/dhcp/issues

exiting.
dhclient(1742) is already running - exiting. 

This version of ISC DHCP is based on the release available
on ftp.isc.org. Features have been added and other changes
have been made to the base software release in order to make
it work better with this distribution.

Please report issues with this software via: 
https://gitee.com/src-openeuler/dhcp/issues

exiting.
[ceph-2][DEBUG ] connected to host: ceph-2 
[ceph-2][DEBUG ] detect platform information from remote host
21.10U3 LTS
bclinux
[ceph-2][DEBUG ] detect machine type
[ceph_deploy.rgw][INFO  ] Distro info: bclinux 21.10U3 21.10U3 LTS
[ceph_deploy.rgw][DEBUG ] remote host will use systemd
[ceph_deploy.rgw][DEBUG ] deploying rgw bootstrap to ceph-2
[ceph-2][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph-2][WARNIN] rgw keyring does not exist yet, creating one
[ceph-2][DEBUG ] create a keyring file
[ceph-2][DEBUG ] create path recursively if it doesn't exist
[ceph-2][INFO  ] Running command: ceph --cluster ceph --name client.bootstrap-rgw --keyring /var/lib/ceph/bootstrap-rgw/ceph.keyring auth get-or-create client.rgw.ceph-2 osd allow rwx mon allow rw -o /var/lib/ceph/radosgw/ceph-rgw.ceph-2/keyring
[ceph-2][INFO  ] Running command: systemctl enable ceph-radosgw@rgw.ceph-2
[ceph-2][WARNIN] Created symlink /etc/systemd/system/ceph-radosgw.target.wants/ceph-radosgw@rgw.ceph-2.service → /usr/lib/systemd/system/ceph-radosgw@.service.
[ceph-2][INFO  ] Running command: systemctl start ceph-radosgw@rgw.ceph-2
[ceph-2][INFO  ] Running command: systemctl enable ceph.target
[ceph_deploy.rgw][INFO  ] The Ceph Object Gateway (RGW) is now running on host ceph-2 and default port 7480
dhclient(1722) is already running - exiting. 

This version of ISC DHCP is based on the release available
on ftp.isc.org. Features have been added and other changes
have been made to the base software release in order to make
it work better with this distribution.

Please report issues with this software via: 
https://gitee.com/src-openeuler/dhcp/issues

exiting.
dhclient(1722) is already running - exiting. 

This version of ISC DHCP is based on the release available
on ftp.isc.org. Features have been added and other changes
have been made to the base software release in order to make
it work better with this distribution.

Please report issues with this software via: 
https://gitee.com/src-openeuler/dhcp/issues

exiting.
[ceph-3][DEBUG ] connected to host: ceph-3 
[ceph-3][DEBUG ] detect platform information from remote host
21.10U3 LTS
bclinux
[ceph-3][DEBUG ] detect machine type
[ceph_deploy.rgw][INFO  ] Distro info: bclinux 21.10U3 21.10U3 LTS
[ceph_deploy.rgw][DEBUG ] remote host will use systemd
[ceph_deploy.rgw][DEBUG ] deploying rgw bootstrap to ceph-3
[ceph-3][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph-3][WARNIN] rgw keyring does not exist yet, creating one
[ceph-3][DEBUG ] create a keyring file
[ceph-3][DEBUG ] create path recursively if it doesn't exist
[ceph-3][INFO  ] Running command: ceph --cluster ceph --name client.bootstrap-rgw --keyring /var/lib/ceph/bootstrap-rgw/ceph.keyring auth get-or-create client.rgw.ceph-3 osd allow rwx mon allow rw -o /var/lib/ceph/radosgw/ceph-rgw.ceph-3/keyring
[ceph-3][INFO  ] Running command: systemctl enable ceph-radosgw@rgw.ceph-3
[ceph-3][WARNIN] Created symlink /etc/systemd/system/ceph-radosgw.target.wants/ceph-radosgw@rgw.ceph-3.service → /usr/lib/systemd/system/ceph-radosgw@.service.
[ceph-3][INFO  ] Running command: systemctl start ceph-radosgw@rgw.ceph-3
[ceph-3][INFO  ] Running command: systemctl enable ceph.target
[ceph_deploy.rgw][INFO  ] The Ceph Object Gateway (RGW) is now running on host ceph-3 and default port 7480
[root@ceph-0 ceph]# cp: cannot stat '/tmp/security_updates': No such file or directory
cp: cannot stat '/tmp/security_updates': No such file or directory
cp: cannot stat '/tmp/security_updates': No such file or directory

创建成功

验证服务状态(失败)

日志目录

/var/log/ceph

日志文件

ceph-client.rgw.ceph-0.log

清空日志文件,重启服务

cd /var/log/ceph
echo "" > ceph-client.rgw.ceph-0.log
systemctl start ceph-radosgw@rgw-ceph-0

先看服务日志

journalctl -xeu ceph-radosgw@rgw-ceph-0

日志

Nov 14 13:38:08 ceph-0 radosgw[58375]: 2023-11-14 13:38:08.450 ffff7ab37010 -1 auth: unable to find a keyring on /var/lib/ceph/radosgw/ceph-rgw-ceph-0/keyring: (2) No such file or directory

Nov 14 13:38:08 ceph-0 radosgw[58375]: 2023-11-14 13:38:08.450 ffff7ab37010 -1 AuthRegistry(0xffff7a330148) no keyring found at /var/lib/ceph/radosgw/ceph-rgw-ceph-0/keyring, disabling cephx

Nov 14 13:38:08 ceph-0 radosgw[58375]: 2023-11-14 13:38:08.450 ffff7ab37010 -1 auth: unable to find a keyring on /var/lib/ceph/radosgw/ceph-rgw-ceph-0/keyring: (2) No such file or directory

Nov 14 13:38:08 ceph-0 radosgw[58375]: 2023-11-14 13:38:08.450 ffff7ab37010 -1 AuthRegistry(0xffffdc7f98c8) no keyring found at /var/lib/ceph/radosgw/ceph-rgw-ceph-0/keyring, disabling cephx

Nov 14 13:38:08 ceph-0 radosgw[58375]: failed to fetch mon config (--no-mon-config to skip)

Nov 14 13:38:08 ceph-0 systemd[1]: ceph-radosgw@rgw-ceph-0.service: Main process exited, code=exited, status=1/FAILURE

没有按照官方文档执行网关安装有关?ceph-deploy install --rgw node..

但是这个版本的ceph-deploy指定的ceph版本太低,且是el6

解决 /var/lib/ceph/radosgw/ceph-rgw-ceph-0/keyring: (2) No such file or directory

查看/etc/ceph目录下的keyring文件,发现已生成了ceph.bootstrap-rgw.keyring

错误的路径/var/lib/ceph/radosgw/ceph-rgw-ceph-0/keyring

正确的路径/var/lib/ceph/radosgw/ceph-rgw.ceph-0/keyring

不知道为什么路径有一个符号错误

做个软连接

cd /var/lib/ceph/radosgw
ln -s ceph-rgw.ceph-0 ceph-rgw-ceph-0

错误:

Nov 14 14:11:43 ceph-0 radosgw[59264]: 2023-11-14 14:11:43.076 ffff99055730 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2]

Nov 14 14:11:43 ceph-0 radosgw[59264]: 2023-11-14 14:11:43.080 ffff99856730 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2]

Nov 14 14:11:43 ceph-0 radosgw[59264]: failed to fetch mon config (--no-mon-config to skip)

Nov 14 14:11:43 ceph-0 systemd[1]: ceph-radosgw@rgw-ceph-0.service: Main process exited, code=exited, status=1/FAILURE

重新执行部署

cd /etc/ceph
ceph-deploy rgw create ceph-0 ceph-1 ceph-2 ceph-3

重新部署后,会生成/var/log/ceph/ceph-client.rgw.ceph-0.log日志文件

其中有日志

Numerical result out of range

2023-11-14 14:32:24.461 ffff8abe8010 0 deferred set uid:gid to 1000:1000 (ceph:ceph)

2023-11-14 14:32:24.461 ffff8abe8010 0 ceph version 14.2.10 (b340acf629a010a74d90da5782a2c5fe0b54ac20) nautilus (stable), process radosgw, pid 60035

2023-11-14 14:32:25.509 ffff8abe8010 0 rgw_init_ioctx ERROR: librados::Rados::pool_create returned (34) Numerical result out of range (this can be due to a pool or placement group misconfiguration, e.g. pg_num < pgp_num or mon_max_pg_per_osd exceeded)

2023-11-14 14:32:25.509 ffff8abe8010 0 failed reading realm info: ret -34 (34) Numerical result out of range

2023-11-14 14:32:25.509 ffff8abe8010 0 ERROR: failed to start notify service ((34) Numerical result out of range

2023-11-14 14:32:25.509 ffff8abe8010 0 ERROR: failed to init services (ret=(34) Numerical result out of range)

2023-11-14 14:32:25.513 ffff8abe8010 -1 Couldn't init storage provider (RADOS)

设置默认pg_num pgs_num

/etc/ceph/ceph.conf

osd pool default size = 3
osd pool default min_size = 1
osd pool default pg num = 128
osd pool default pgp num = 128

重新部署(--overwrite-conf)

cd /etc/ceph
ceph-deploy --overwrite-conf rgw create ceph-0 ceph-1 ceph-2 ceph-3

故障依旧

成功的方案

删掉所有pool,重新部署,参考以下历史记录

652 ceph osd pool ls

653 ceph osd pool rm vdbench

654 ceph osd pool rm vdbench vdbench --yes-i-really-really-mean-it

655 ls /dev/

656 mount

657 df -h

658 umount /mnt/rbd0

659 ceph osd ls

660 ceph osd pool ls

661 ceph osd volume ls

662 ceph osd dump

663 ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it

664 ceph fs ls

665 ceph fs rm cephfs

666 ceph fs fail

667 ceph fs fail cephfs

668 ceph -s

669 ceph fs status cephfs

670 ceph fs ls

671 ceph fs rm cephfs

672 ceph fs rm cephfs --yes-i-really-mean-it

673 ceph fs ls

674 ceph osd pool ls

675 ceph osd pool rm cephfs_data cephfs_data --yes-i-really-mean-it

676 ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it

677 ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it

678 ceph -

679 ceph -s

680 cd /etc/ceph

681 ceph-deploy rgw create ceph-0

682 cat /var/log/ceph/ceph-client.rgw.ceph-0.log

683 systemctl status ceph-radosgw@rgw.ceph-0

再来一次集群环境的部署

cd /etc/ceph
ceph-deploy --overwrite-conf rgw create ceph-0 ceph-1 ceph-2 ceph-3

部分日志

ceph -s rgw已经起来了

======================

以下为参考官方文档的失败经理,留档参考

部署对象存储网关rgw(失败)

ceph-deploy install --rgw ceph-0 ceph-1 ceph-2 ceph-3

故障 Unable to find a match: epel-release

dnf install https://dl.fedoraproject.org/pub/epel/epel-release-latest-8.noarch.rpm -y

故障Status code: 404

坑爹的阿里云镜像,没有epel 8,bclinux又是el8版本!

采用huawei epel镜像

[epel-everything]
name=huawei epel mirror
baseurl=https://repo.huaweicloud.com/epel/8/Everything/aarch64/
enabled=1
gpgcheck=0

yum install epel-release --nobest --skip-broken

ceph-deploy 会强制执行yum install epel-release(失败)

注释掉epel-release的安装

修改/usr/lib/python2.7/site-packages/ceph_deploy/hosts/centos/install.py 60行,注释掉epel-release的安装(前面已经手动--skip-broken安装好了)

生效了

故障 Unable to find a match: yum-plugin-priorities

参考上面思路,直接注释掉/usr/lib/python2.7/site-packages/ceph_deploy/hosts/centos/install.py 61,62行

自动找到老版本ceph-release-1-0.el6 Status code: 404

手动安装

修改/usr/lib/python2.7/site-packages/ceph_deploy/install.py

安装版本参数args.release = 'mimic' 改为 args.release = 'nautilus

关于python3版本

bclinux python3.7.9

cenos7 python3.6.8

openeuler 20.03 lts sp3 python 3.7.9

参考

Ceph Object Gateway --- Ceph Documentation

如何在 CentOS 8 和 RHEL 8 服务器上启用 EPEL 仓库 - 知乎 (zhihu.com)

ceph-deploy部署ceph-nautilus_ceph nautilus-CSDN博客

相关推荐
菜菜-plus2 小时前
分布式,微服务,SpringCloudAlibaba,nacos,gateway,openFeign
java·分布式·微服务·nacos·gateway·springcloud·openfeign
七月在野,八月在宇,九月在户2 小时前
前端--> nginx-->gateway产生的跨域问题分析
前端·nginx·gateway
墨水\\2 天前
分布式----Ceph部署(上)
分布式·ceph
Denny辉2 天前
Flink使用SQL Gateway提交SQL Job到远程集群
sql·flink·gateway
菜菜-plus2 天前
Nginx、Gateway的区别
运维·nginx·gateway
cyt涛4 天前
SpringCloudGateway — 网关登录校验
运维·网关·gateway·登录·过滤器·校验·threadlocal
牛角上的男孩6 天前
Istio Gateway发布服务
云原生·gateway·istio
斯普信专业组8 天前
CephFS管理秘籍:全面掌握文件系统与MDS守护程序命令
ceph·cephfs
柳叶寒8 天前
医院信息化与智能化系统(17)
java·nacos·gateway·全栈·项目
cyt涛8 天前
SpringCloudGateway — 网关路由
java·开发语言·网关·gateway·路由·断言·转发