docker gitlab14.4.2-14.10.5升级过程记录

1 背景

公安局来检查,并出具了 <XXX分局网络安全监督检查限期整改通知书>,因gitlab版本较旧,存在两个高危漏洞,所以让升级.

2 漏洞说明

代码管理服务器(gitlab)总计检测出两个漏洞分别是

cve-2022-0735

cve-2022-2185

2.1 cve-2022-0735

2.1.1 漏洞描述

An issue has been discovered in GitLab CE/EE affecting all versions starting from 12.10 before 14.6.5, all versions starting from 14.7 before 14.7.4, all versions starting from 14.8 before 14.8.2. An unauthorized user was able to steal runner registration tokens through an information disclosure vulnerability using quick actions commands. This is a critical severity issue (CVSS:3.0/AV:N/AC:L/PR:L/UI:N/S:C/C:H/I:H/A:N, 9.6). It is now mitigated in the latest release and is assigned CVE-2022-0735.

2.1.2 涉及版本

从 12.10 开始到 14.6.5 之前的所有版本

从 14.7 开始到 14.7.4 之前的所有版本

从 14.8 开始到 14.8.2 之前的所有版本

2.1.3 安全版本

GitLab CE/EE 14.8.2;

GitLab CE/EE 14.7.4;

GitLab CE/EE 14.6.5

2.2 cve-2022-2185

2.2.1 漏洞描述

Remote Command Execution via Project Imports

A critical issue has been discovered in GitLab affecting all versions starting from 14.0 prior to 14.10.5, 15.0 prior to 15.0.4, and 15.1 prior to 15.1.1 where an authenticated user authorized to import projects could import a maliciously crafted project leading to remote code execution.

This is a critical severity issue (CVSS:3.1/AV:N/AC:L/PR:L/UI:N/S:C/C:H/I:H/A:H, 9.9). It is now mitigated in the latest release and is assigned CVE-2022-2185.

2.2.2 涉及版本

GitLab CE/EE 14.0 版本：< 14.10.5
GitLab CE/EE 15.0 版本：< 15.0.4
GitLab CE/EE 15.1 版本：< 15.1.1

2.2.3 安全版本

GitLab CE/EE 14.10.5
GitLab CE/EE 15.0.4
GitLab CE/EE 15.1.1

3 升级过程

3.1 升级路径确定

参考官网的升级路线https://archives.docs.gitlab.com/15.11/ee/update/#upgrade-paths

我们服务器当前版本为14.4.2

我们服务器的升级路线为

	14.4.2 > 14.9.5 > 14.10.5

3.2 当前服务器状态

3.2.1 系统版本

DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=18.04
DISTRIB_CODENAME=bionic
DISTRIB_DESCRIPTION="Ubuntu 18.04.6 LTS"

3.2.2 Gitlab安装方式和版本

安装方式为docker容器安装

版本为14.4.2

3.3 升级前数据备份

将gitlab所有内容进行备份

cd /home  
tar -czf /mnt/gitlab/gitlab_backup_14.4.2_$(date '+%Y%m%d%H%M%S').tar.gz gitlab/  

3.4 Gitlab 14.4.2升级到14.9.5-ce.0

3.4.1 停止容器运行

输入

docker ps  

输出

CONTAINER ID   IMAGE          COMMAND             CREATED       STATUS                       PORTS                                                                           NAMES  
55cabaf20c60   7b8db4329c1c   "/assets/wrapper"   2 years ago   Up About an hour (healthy)   0.0.0.0:22->22/tcp, :::22->22/tcp, 0.0.0.0:80->80/tcp, :::80->80/tcp, 443/tcp   gitlab  

输入

docker stop gitlab   

3.4.2 备份当前docker镜像文件

docker images  

输出

REPOSITORY         TAG       IMAGE ID       CREATED         SIZE  
gitlab/gitlab-ce   latest    7b8db4329c1c   2 years ago     2.31GB  

输入

docker save -o gitlab-ce_14.4.2.tar gitlab/gitlab-ce:latest  

查看

ls -lh gitlab-ce_14.4.2.tar  
-rw------- 1 root root 2.3G Dec 13 15:53 gitlab-ce_14.4.2.tar  

3.4.3 下载14.9.5-ce.0镜像

输入

docker pull gitlab/gitlab-ce:14.9.5-ce.0

返回

1.	14.9.5-ce.0: Pulling from gitlab/gitlab-ce  
2.	d5fd17ec1767: Pull complete   
3.	c73908ed6492: Pull complete   
4.	588bf1079275: Pull complete   
5.	79d508e4cd50: Pull complete   
6.	680e6c2e4367: Pull complete   
7.	ae617a72a43f: Pull complete   
8.	55fb226a44ad: Pull complete   
9.	c9990399c59f: Pull complete   
10.	Digest: sha256:6261498881c53dc95a0c4784fcd931ce37b1cfd3276a5e2a4cdb1fd45a1594f9  
11.	Status: Downloaded newer image for gitlab/gitlab-ce:14.9.5-ce.0  
12.	docker.io/gitlab/gitlab-ce:14.9.5-ce.0  

3.4.4 创建并启动容器

在旧版本数据基础上启动容器,新的容器名称为gitlab-14.9.5-ce.0

docker run \
  -itd  \
  -p 80:80 \
  -p 22:22 \
  -v /home/gitlab/config:/etc/gitlab \
  -v /home/gitlab/logs:/var/log/gitlab \
  -v /home/gitlab/data:/var/opt/gitlab \
  --restart always \
  --privileged=true \
  --name gitlab-14.9.5-ce.0 \
  gitlab/gitlab-ce:14.9.5-ce.0

3.4.5 升级后的修复处理

1. 官网原文参考
   

2. 进入控制台

   docker exec -it <container-id> gitlab-rails console

3. 执行数据迁移命令

   Gitlab::Database::BackgroundMigrationJob.pending.where(class_name: "ResetDuplicateCiRunnersTokenValuesOnProjects").find_each do |job| puts Gitlab::Database::BackgroundMigrationJob.mark_all_as_succeeded("ResetDuplicateCiRunnersTokenValuesOnProjects", job.arguments)
   end

3.4.6 登陆查看

3.5 Gitlab 14.9.5-ce.0升级到14.10.5-ce.0

3.5.1 停止容器

docker stop gitlab-14.9.5-ce.0   

3.5.2 下载14.10.5-ce.0镜像

输入

docker pull gitlab/gitlab-ce:14.10.5-ce.0 

返回

  14.10.5-ce.0: Pulling from gitlab/gitlab-ce  
  d7bfe07ed847: Pull complete   
  36eab9ae4aa1: Pull complete   
  10f53ed78fe9: Pull complete   
  975e6c523eb3: Pull complete   
  cfd1332509fe: Pull complete   
  7d80bd5dd16d: Pull complete   
  3f23f1d159b8: Pull complete   
  964197f2d9b2: Pull complete   
  Digest: sha256:28fb063701b7238ef10fb002da88e677308f85d823435dba9da0223703076a6c  
  Status: Downloaded newer image for gitlab/gitlab-ce:14.10.5-ce.0  
  docker.io/gitlab/gitlab-ce:14.10.5-ce.0  

查看

>>> docker images  
  REPOSITORY         TAG            IMAGE ID       CREATED         SIZE  
  gitlab/gitlab-ce   14.10.5-ce.0   3d036870c870   17 months ago   2.46GB  
  gitlab/gitlab-ce   14.9.5-ce.0    c99393c4ee7f   18 months ago   2.45GB  
  gitlab/gitlab-ce   14.4.2         7b8db4329c1c   2 years ago     2.31GB 

3.5.3 创建并启动容器

  >>> docker run \
   -itd  \
   -p 80:80 \
   -p 22:22 \
   -v /home/gitlab/config:/etc/gitlab \
   -v /home/gitlab/logs:/var/log/gitlab \
   -v /home/gitlab/data:/var/opt/gitlab \
   --restart always \
   --privileged=true \
   --name gitlab-14.10.5-ce.0 \
   gitlab/gitlab-ce:14.10.5-ce.0 

3.5.4 升级后修复处理

1. 官网原文参考
   

   docker exec -it 4bcff587e247 bash
   root@4bcff587e247:/# gitlab-rake db:migrate
   root@4bcff587e247:/# gitlab-ctl reconfigure
   root@4bcff587e247:/# gitlab-ctl hup puma
   root@4bcff587e247:/# gitlab-ctl restart sidekiq

3.5.5 查看容器状态

>>> docker ps --no-trunc   
CONTAINER ID                                                       IMAGE                           COMMAND             CREATED         STATUS                   PORTS                                                                           NAMES  
4bcff587e2470f7650f8c86b93f2e6beeb2c3e21720505f782f7725243f37046   gitlab/gitlab-ce:14.10.5-ce.0   "/assets/wrapper"   9 minutes ago   Up 9 minutes (healthy)   0.0.0.0:22->22/tcp, :::22->22/tcp, 0.0.0.0:80->80/tcp, :::80->80/tcp, 443/tcp   gitlab-14.10.5-ce.0  

3.5.6 登陆查看

4 收尾

4.1 服务器状态检测脚本更新

docker-gitlab-check.sh

#! /bin/sh

CONTAINER_NAME="gitlab-14.10.5-ce.0"
ps_res=$(docker ps -a | grep "${CONTAINER_NAME}")


if echo $ps_res | grep -w -E "health|healthy" > /dev/null
then
 status="healthy"
else
 status="unhealthy"
fi

logger --id=${PPID} -t gitlab_check -p daemon.info "${CONTAINER_NAME} : status=$status $ps_res"


if [ "$status" = "unhealthy" ]
then
 logger --id=${PPID} -t gitlab_check -p daemon.warning "run :docker stop ${CONTAINER_NAME}"
 docker stop ${CONTAINER_NAME}
 # docker gitlab reused sshd

 logger --id=${PPID} -t gitlab_check -p daemon.warning "run :service sshd stop"
 service sshd stop

 logger --id=${PPID} -t gitlab_check -p daemon.warning "run :docker start ${CONTAINER_NAME}"
 docker start ${CONTAINER_NAME}

 logger --id=${PPID} -t gitlab_check -p daemon.warning "restart docker ${CONTAINER_NAME}"
fi

4.2 备份脚本更新

docker-gitlab-backup.sh

#! /bin/sh
# needed root

CONTAINER_NAME="gitlab-14.10.5-ce.0"
TIMESTAMP=$(date '+%Y_%m_%d_%s')
BACKUPFILE=${TIMESTAMP}_gitlab_backup
docker exec -it ${CONTAINER_NAME} gitlab-backup create GZIP_RSYNCABLE=yes BACKUP=${TIMESTAMP}
# touch /home/gitlab/data/backups/${BACKUPFILE}.tar

# file is ${TIMESTAMP}_gitlab_backup.tar
if [ -f /home/gitlab/data/backups/${BACKUPFILE}.tar ];then 
        cd /home/gitlab ;
        tar -czf /mnt/gitlab_backup/${BACKUPFILE}.tar.gz config/ logs/ data/backups/${BACKUPFILE}.tar && rm data/backups/${BACKUPFILE}.tar;
        cd -
        logger --id=${PPID} -t gitlab_backup -p daemon.warning "backup ${CONTAINER_NAME} /mnt/gitlab_backup/${BACKUPFILE}.tar.gz"
fi

4.3 定时配置

/etc/cron.d/docker-gitlab

1.	SHELL=/bin/sh
2.	PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
3.	
4.	# For details see man 4 crontabs
5.	
6.	# Example of job definition:
7.	# .---------------- minute (0 - 59)
8.	# |  .------------- hour (0 - 23)
9.	# |  |  .---------- day of month (1 - 31)
10.	# |  |  |  .------- month (1 - 12) OR jan,feb,mar,apr ...
11.	# |  |  |  |  .---- day of week (0 - 6) (Sunday=0 or 7) OR sun,mon,tue,wed,thu,fri,sat
12.	# |  |  |  |  |
13.	# *  *  *  *  * user-name  command to be executed
14.	
15.	*/10 *  *  *  * root  sh /home/serveradmin/docker-gitlab-check.sh > /dev/null
16.	  0  0  1  *  * root  sh /home/serveradmin/docker-gitlab-backup.sh

4.4 旧版本容器和镜像清理

注意:当前版本运行一周后进行清理

现有容器信息

1.	>>> docker ps --format '{{ .ID }}\t{{ .Names }}' -a              
2.	4bcff587e247    gitlab-14.10.5-ce.0  
3.	fa9c46b86a8a    gitlab-14.9.5-ce.0  
4.	55cabaf20c60    gitlab  

删除容器

docker rm 55cabaf20c60 fa9c46b86a8a

现有镜像信息

1.	>>> docker images   
2.	REPOSITORY         TAG            IMAGE ID       CREATED         SIZE  
3.	gitlab/gitlab-ce   14.10.5-ce.0   3d036870c870   17 months ago   2.46GB  
4.	gitlab/gitlab-ce   14.9.5-ce.0    c99393c4ee7f   18 months ago   2.45GB  
5.	gitlab/gitlab-ce   14.4.2         7b8db4329c1c   2 years ago     2.31GB 

删除旧版本镜像

1.	>>> docker rmi gitlab/gitlab-ce:14.4.2 gitlab/gitlab-ce:14.9.5-ce.0