【全网独家】oceanbase容器重启时报obshell failed错误,无法正常启动的问题处理

正常运行的oceanbase容器,重新启动该容器却启动不了,重启服务器也无法恢复,报obshell failed错误,无法正常启动,本文记录了问题处理过程。

一、问题现象

1、正常运行的oceanbase容器,重启却启动不了

2、运行docker logs oceanbase检查日志,出错信息如下

核心错误为以下两句

[ERROR] 127.0.0.1 obshell failed
[ERROR] oceanbase-ce start failed

并提示运行 "obd display-trace 3d1c71c4-f80a-11ee-947f-0242ac110002"来检查obd的日志信息。

二、问题分析

1、定位问题

此时容器已无法启动,无法进入容器运行obd display-trace命令,但还好数据目录是挂载的主机目录 /app/dockerdata/oceanbase/obd,相应日志文件在主机侧可以直接查看。

bash 复制代码
[root@localhost ~]# cat /app/dockerdata/oceanbase/obd/log/obd 
....
[2024-04-11 13:48:56.356] [3d1c71c4-f80a-11ee-947f-0242ac110002] [DEBUG] -- exited code 2, error output:
[2024-04-11 13:48:56.356] [3d1c71c4-f80a-11ee-947f-0242ac110002] [DEBUG] ls: cannot access '/proc/118': No such file or directory
[2024-04-11 13:48:56.356] [3d1c71c4-f80a-11ee-947f-0242ac110002] [DEBUG] 
[2024-04-11 13:48:56.356] [3d1c71c4-f80a-11ee-947f-0242ac110002] [DEBUG] -- root@127.0.0.1 set env OB_ROOT_PASSWORD to ''
[2024-04-11 13:48:56.356] [3d1c71c4-f80a-11ee-947f-0242ac110002] [DEBUG] -- start obshell: cd /root/ob; /root/ob/bin/obshell admin start --ip 127.0.0.1 --port 2886
[2024-04-11 13:48:56.356] [3d1c71c4-f80a-11ee-947f-0242ac110002] [DEBUG] -- local execute: cd /root/ob; /root/ob/bin/obshell admin start --ip 127.0.0.1 --port 2886 
[2024-04-11 13:48:57.414] [3d1c71c4-f80a-11ee-947f-0242ac110002] [DEBUG] -- exited code 29, error output:
[2024-04-11 13:48:57.415] [3d1c71c4-f80a-11ee-947f-0242ac110002] [DEBUG] open /root/ob/run/daemon.pid: file exists
[2024-04-11 13:48:57.415] [3d1c71c4-f80a-11ee-947f-0242ac110002] [DEBUG] 
[2024-04-11 13:48:57.415] [3d1c71c4-f80a-11ee-947f-0242ac110002] [ERROR] 127.0.0.1 obshell failed
[2024-04-11 13:48:57.416] [3d1c71c4-f80a-11ee-947f-0242ac110002] [DEBUG] - sub start ref count to 0
[2024-04-11 13:48:57.416] [3d1c71c4-f80a-11ee-947f-0242ac110002] [DEBUG] - export start
[2024-04-11 13:48:57.416] [3d1c71c4-f80a-11ee-947f-0242ac110002] [ERROR] oceanbase-ce start failed
[2024-04-11 13:48:57.420] [3d1c71c4-f80a-11ee-947f-0242ac110002] [INFO] See https://www.oceanbase.com/product/ob-deployer/error-codes .
[2024-04-11 13:48:57.420] [3d1c71c4-f80a-11ee-947f-0242ac110002] [INFO] Trace ID: 3d1c71c4-f80a-11ee-947f-0242ac110002
[2024-04-11 13:48:57.420] [3d1c71c4-f80a-11ee-947f-0242ac110002] [INFO] If you want to view detailed obd logs, please run: obd display-trace 3d1c71c4-f80a-11ee-947f-0242ac110002
[2024-04-11 13:48:57.421] [3d1c71c4-f80a-11ee-947f-0242ac110002] [DEBUG] - share lock /root/.obd/lock/mirror_and_repo release, count 1
[2024-04-11 13:48:57.421] [3d1c71c4-f80a-11ee-947f-0242ac110002] [DEBUG] - share lock /root/.obd/lock/mirror_and_repo release, count 0
[2024-04-11 13:48:57.421] [3d1c71c4-f80a-11ee-947f-0242ac110002] [DEBUG] - unlock /root/.obd/lock/mirror_and_repo
[2024-04-11 13:48:57.421] [3d1c71c4-f80a-11ee-947f-0242ac110002] [DEBUG] - exclusive lock /root/.obd/lock/deploy_obcluster release, count 0
[2024-04-11 13:48:57.421] [3d1c71c4-f80a-11ee-947f-0242ac110002] [DEBUG] - unlock /root/.obd/lock/deploy_obcluster
[2024-04-11 13:48:57.421] [3d1c71c4-f80a-11ee-947f-0242ac110002] [DEBUG] - share lock /root/.obd/lock/global release, count 0
[2024-04-11 13:48:57.421] [3d1c71c4-f80a-11ee-947f-0242ac110002] [DEBUG] - unlock /root/.obd/lock/global

可以看到关键的出错信息为:

[2024-04-11 13:48:57.415] [3d1c71c4-f80a-11ee-947f-0242ac110002] [DEBUG] open /root/ob/run/daemon.pid: file exists

[2024-04-11 13:48:57.415] [3d1c71c4-f80a-11ee-947f-0242ac110002] [DEBUG]

[2024-04-11 13:48:57.415] [3d1c71c4-f80a-11ee-947f-0242ac110002] [ERROR] 127.0.0.1 obshell failed

[2024-04-11 13:48:57.416] [3d1c71c4-f80a-11ee-947f-0242ac110002] [DEBUG] - sub start ref count to 0

[2024-04-11 13:48:57.416] [3d1c71c4-f80a-11ee-947f-0242ac110002] [DEBUG] - export start

[2024-04-11 13:48:57.416] [3d1c71c4-f80a-11ee-947f-0242ac110002] [ERROR] oceanbase-ce start failed

即容器在启动ob时发现/root/ob/run/daemon.pid存在,认为程序仍在运行退出,随即obshell 启动失败,导致最后oceanbase-ce启动失败。

三、解决办法

容器内的/root/ob/run/daemon.pid对应主机/app/dockerdata/oceanbase/ob/run/daemon.pid,察看文件内容

bash 复制代码
[root@localhost ~]# cat /app/dockerdata/oceanbase/ob/run/daemon.pid
98

里面的值为上次容器运行时守护进程的pid,删除该文件,重启容器

bash 复制代码
[root@localhost ~]# rm /app/dockerdata/oceanbase/ob/run/daemon.pid
rm: remove regular file '/app/dockerdata/oceanbase/ob/run/daemon.pid'? y
[root@localhost ~]# docker restart oceanbase
oceanbase
[root@localhost ~]# docker ps -a
CONTAINER ID        IMAGE                    COMMAND              CREATED             STATUS              PORTS                    NAMES
e2f1998af148        oceanbase/oceanbase-ce   "/bin/sh -c _boot"   38 minutes ago      Up 6 seconds        0.0.0.0:3306->2881/tcp   oceanbase

容器恢复正常 ,尝试登录:

bash 复制代码
[root@localhost ~]# mysql -h127.0.0.1 -uroot -p -P3306
Enter password: 
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 3221487687
Server version: 5.7.25 OceanBase_CE 4.3.0.1 (r100000242024032211-0193a343bc60b4699ec47792c3fc4ce166a182f9) (Built Mar 22 2024 13:19:48)

Copyright (c) 2000, 2022, Oracle and/or its affiliates.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| LBACSYS            |
| mysql              |
| oceanbase          |
| ocs                |
| ORAAUDITOR         |
| SYS                |
| test               |
+--------------------+
8 rows in set (0.02 sec)

mysql> exit
Bye
[root@localhost ~]# 

可见业务已经恢复。

经查,这是oceanbase容器的一个运行BUG,通过docker restart oceanbase(oceanbase为运行的容器名)就必然会启不来了,要删掉pid文件才能重新正常启动,:-(。

相关推荐
wclass-zhengge6 分钟前
Docker篇(Docker Compose)
运维·docker·容器
knight-n10 分钟前
MYSQL库的操作
数据库·mysql
李启柱14 分钟前
项目开发流程规范文档
运维·软件构建·个人开发·设计规范
包饭厅咸鱼1 小时前
QML----复制指定下标的ListModel数据
开发语言·数据库
生命几十年3万天1 小时前
redis时间优化
数据库·redis·缓存
Elastic 中国社区官方博客1 小时前
释放专利力量:Patently 如何利用向量搜索和 NLP 简化协作
大数据·数据库·人工智能·elasticsearch·搜索引擎·自然语言处理
力姆泰克1 小时前
看电动缸是如何提高农机的自动化水平
大数据·运维·服务器·数据库·人工智能·自动化·1024程序员节
力姆泰克1 小时前
力姆泰克电动缸助力农业机械装备,提高农机的自动化水平
大数据·服务器·数据库·人工智能·1024程序员节
Lionhacker1 小时前
网络工程师这个行业可以一直干到退休吗?
网络·数据库·网络安全·黑客·黑客技术
BPM_宏天低代码2 小时前
低代码 BPA:简化业务流程自动化的新趋势
运维·低代码·自动化