20240622服务器异常

异常表现

玩法无法登录服务器，远程XShell 无法连接到服务器,IT 后台查看服务器内存使用完毕。

异常处理

IT 直接重启服务器

后续异常

Docker 服务器无法启动

• 使用命令查看错误：

  sudo systemctl status docker.service

× docker.service - Docker Application Container Engine

Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled; preset: disabled)

Active: failed (Result: exit-code) since Sun 2024-06-23 00:49:22 CST; 1s ago

TriggeredBy: × docker.socket

Docs: https://docs.docker.com

Process: 3848 ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock (code=exited, status=1/FAILURE)

Main PID: 3848 (code=exited, status=1/FAILURE)

CPU: 222ms

Jun 23 00:49:22 iZj6c4nk0afv2j81vxzb4rZ systemd[1]: docker.service: Scheduled restart job, restart counter is at 3.

Jun 23 00:49:22 iZj6c4nk0afv2j81vxzb4rZ systemd[1]: Stopped Docker Application Container Engine.

Jun 23 00:49:22 iZj6c4nk0afv2j81vxzb4rZ systemd[1]: docker.service: Start request repeated too quickly.

Jun 23 00:49:22 iZj6c4nk0afv2j81vxzb4rZ systemd[1]: docker.service: Failed with result 'exit-code'.

• 使用命令查看错误：

  journalctl -u docker

Jun 23 01:10:31 iZj6c4nk0afv2j81vxzb4rZ systemd[1]: docker.service: Scheduled restart job, restart counter is at 2.

Jun 23 01:10:31 iZj6c4nk0afv2j81vxzb4rZ systemd[1]: Stopped Docker Application Container Engine.

Jun 23 01:10:31 iZj6c4nk0afv2j81vxzb4rZ systemd[1]: Starting Docker Application Container Engine...

Jun 23 01:10:31 iZj6c4nk0afv2j81vxzb4rZ dockerd[4956]: time="2024-06-23T01:10:31.864915773+08:00" level=info msg="Starting up"

Jun 23 01:10:31 iZj6c4nk0afv2j81vxzb4rZ dockerd[4956]: time="2024-06-23T01:10:31.889762083+08:00" level=info msg="[graphdriver] using prior storage driver: overlay2"

Jun 23 01:10:31 iZj6c4nk0afv2j81vxzb4rZ dockerd[4956]: time="2024-06-23T01:10:31.902923375+08:00" level=info msg="Loading containers: start."

Jun 23 01:10:32 iZj6c4nk0afv2j81vxzb4rZ dockerd[4956]: time="2024-06-23T01:10:32.096278048+08:00" level=warning msg="could not create bridge network for id 37fd119fe2ef5e1ffc8be243920dcf15e08353b7802e1537ac2669f652252e25 bridge name docker0>

Jun 23 01:10:32 iZj6c4nk0afv2j81vxzb4rZ dockerd[4956]: time="2024-06-23T01:10:32.133241662+08:00" level=warning msg="could not create bridge network for id be34da5caf5b3546f5bc3f599729295c12ce8a5a6f148b0edddc8fee6b4374fa bridge name br-be34>

Jun 23 01:10:32 iZj6c4nk0afv2j81vxzb4rZ dockerd[4956]: time="2024-06-23T01:10:32.166272050+08:00" level=warning msg="could not create bridge network for id c123ffb11dc79085025ae7a443bda6232534918c7988d1e2b4c39cce9473f0da bridge name br-c123>

Jun 23 01:10:32 iZj6c4nk0afv2j81vxzb4rZ dockerd[4956]: time="2024-06-23T01:10:32.170134695+08:00" level=info msg="Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a preferred IP a>

Jun 23 01:10:32 iZj6c4nk0afv2j81vxzb4rZ dockerd[4956]: time="2024-06-23T01:10:32.203367825+08:00" level=info msg="stopping event stream following graceful shutdown" error="<nil>" module=libcontainerd namespace=moby

Jun 23 01:10:32 iZj6c4nk0afv2j81vxzb4rZ dockerd[4956]: failed to start daemon: Error initializing network controller: error creating default "bridge" network: Failed to Setup IP tables: Unable to enable NAT rule: (iptables failed: iptables>

Jun 23 01:10:32 iZj6c4nk0afv2j81vxzb4rZ dockerd[4956]: iptables v1.8.8 (nf_tables): CHAIN_ADD failed (No such file or directory): chain POSTROUTING

Jun 23 01:10:32 iZj6c4nk0afv2j81vxzb4rZ dockerd[4956]: (exit status 4))

Jun 23 01:10:32 iZj6c4nk0afv2j81vxzb4rZ systemd[1]: docker.service: Main process exited, code=exited, status=1/FAILURE

Jun 23 01:10:32 iZj6c4nk0afv2j81vxzb4rZ systemd[1]: docker.service: Failed with result 'exit-code'.

Jun 23 01:10:32 iZj6c4nk0afv2j81vxzb4rZ systemd[1]: Failed to start Docker Application Container Engine.

Jun 23 01:10:34 iZj6c4nk0afv2j81vxzb4rZ systemd[1]: docker.service: Scheduled restart job, restart counter is at 3.

Jun 23 01:10:34 iZj6c4nk0afv2j81vxzb4rZ systemd[1]: Stopped Docker Application Container Engine.

Jun 23 01:10:34 iZj6c4nk0afv2j81vxzb4rZ systemd[1]: docker.service: Start request repeated too quickly.

Jun 23 01:10:34 iZj6c4nk0afv2j81vxzb4rZ systemd[1]: docker.service: Failed with result 'exit-code'.

Jun 23 01:10:34 iZj6c4nk0afv2j81vxzb4rZ systemd[1]: Failed to start Docker Application Container Engine.

Docker无法启动解决方案

• 移除当前docker 的所有网关

  sudo rm -rf /var/lib/docker/network

• 启动docker

  sudo systemctl start docker

• 新建一个网关

因为我们当前有需要继续运行的docker实例，所以需要

docker network create --drver bridge --subnet 192.168.0.0/16 --gateway 192.168.0.1 mynet

• 把新网关连接到 已有的实例当中

  docker network connect mynet game-1

• 启动对应的实例

Docker非正常关闭导致实例启动异常

• MongoDB数据库启动时异常

{"t":{"$date":"2024-06-23T02:20:17.337+00:00"},"s":"I", "c":"CONTROL", "id":51765, "ctx":"initandlisten","msg":"Operating System","attr":{"os":{"name":"Ubuntu","version":"20.04"}}}

{"t":{"$date":"2024-06-23T02:20:17.337+00:00"},"s":"I", "c":"CONTROL", "id":21951, "ctx":"initandlisten","msg":"Options set by command line","attr":{"options":{"net":{"bindIp":"*"}}}}

{"t":{"$date":"2024-06-23T02:20:17.337+00:00"},"s":"E", "c":"NETWORK", "id":23024, "ctx":"initandlisten","msg":"Failed to unlink socket file","attr":{"path":"/tmp/mongodb-27017.sock","error":"Permission denied"}}

{"t":{"$date":"2024-06-23T02:20:17.337+00:00"},"s":"F", "c":"-", "id":23091, "ctx":"initandlisten","msg":"Fatal assertion","attr":{"msgid":40486,"file":"src/mongo/transport/transport_layer_asio.cpp","line":989}}

{"t":{"$date":"2024-06-23T02:20:17.337+00:00"},"s":"F", "c":"-", "id":23092, "ctx":"initandlisten","msg":"\n\n***aborting after fassert() failure\n\n"}

• 找到 MongoDB 在物理机上的存储地址

  docker inspect mongodb

"GraphDriver": {

"Data": {

"LowerDir": "/var/lib/docker/overlay2/4506e9c54c923708d1c59abb0dfe579a8981e5cc895a97d0e109b61d28e525f8-init/diff:/var/lib/docker/overlay2/4999b67d0cfd926f6a79535f4ab58eec3d974598782c32a42b1def3e3f3004c4/diff:/var/lib/docker/overlay2/8f635a5b5cd6e2b3f0c0bacc753ffd82878d653a804caf447b7566b4d9ced6e4/diff:/var/lib/docker/overlay2/2b821c2d9decea1f911b4e19f464a6b4ac612ba9649775a07848f4532db2942c/diff:/var/lib/docker/overlay2/5890e82752bcf368c5eea8cde6005cf1cb1778b454fc106469a90f3e049ce337/diff:/var/lib/docker/overlay2/8ff4f12e4fa6262e456abe1e8602d7b7ddd870ea45a236f1b8329ae4c641f511/diff:/var/lib/docker/overlay2/f17cdb060e6abe3c224b58ca5c4fa2569bbd8ab64f9ea26d818421d317bb3d90/diff:/var/lib/docker/overlay2/45a483a9e0e91027907c15e908e7018292d528e65d40de8815d2e60f574c0caa/diff:/var/lib/docker/overlay2/a41f440c4c1205e009b85d970376353a41e51c0e719005de92c94fc7d4656457/diff:/var/lib/docker/overlay2/b4f1c548715627638cbddea887f48fc7aea9be067e793ea6872337088fa1c864/diff:/var/lib/docker/overlay2/ef7286519ac0028ad63d05b62e68a819de2d448b61b7deabcbd4e5f0aebc792a/diff",

"MergedDir": "/var/lib/docker/overlay2/4506e9c54c923708d1c59abb0dfe579a8981e5cc895a97d0e109b61d28e525f8/merged",

"UpperDir": "/var/lib/docker/overlay2/4506e9c54c923708d1c59abb0dfe579a8981e5cc895a97d0e109b61d28e525f8/diff",

"WorkDir": "/var/lib/docker/overlay2/4506e9c54c923708d1c59abb0dfe579a8981e5cc895a97d0e109b61d28e525f8/work"

},

"Name": "overlay2"

},

• 修改MongoDB 异常

进入：

"UpperDir": "/var/lib/docker/overlay2/4506e9c54c923708d1c59abb0dfe579a8981e5cc895a97d0e109b61d28e525f8/diff",

找到 tmp/mongodb-27017.sock 文件，删除掉，

• 重启 MongoDB异常

Error setting up listener","attr":{"error":{"code":9001,"codeName":"SocketException","errmsg":"Permission denied"}},

• 修改权限问题

找到 docker 镜像和实例所在目录 添加权限

sudo chmod 777 /var/lib/docker

• 重启MongodDB