harbor(docker仓库)仓库部署 - 高可用

harbor(docker仓库)仓库部署 - 高可用

  • [1. harbor高可用](#1. harbor高可用)
    • [1.1 方案说明](#1.1 方案说明)
      • [1. 双主复制](#1. 双主复制)
      • [2. 多harbor实例共享后端存储](#2. 多harbor实例共享后端存储)
    • [1.2 部署高可用(多harbor实例共享后端存储)](#1.2 部署高可用(多harbor实例共享后端存储))
      • [1. 服务器划分](#1. 服务器划分)
      • [2. 安装harbor(先部署一套Harbor,用于将其所有表结构导出)](#2. 安装harbor(先部署一套Harbor,用于将其所有表结构导出))
      • [3. 安装Postgresql](#3. 安装Postgresql)
      • [4. 安装nginx](#4. 安装nginx)
      • [5. 安装nfs](#5. 安装nfs)
      • [6. 安装redis](#6. 安装redis)
      • [7. 部署harbor](#7. 部署harbor)
      • [8. 修改nginx配置](#8. 修改nginx配置)
      • [9. docker登录harbor](#9. docker登录harbor)
      • [10. harbor修改](#10. harbor修改)
      • [11. 修改nginx](#11. 修改nginx)
      • [12. docker推送](#12. docker推送)

1. harbor高可用

目前有两种主流的方案来解决这个问题:

  • 双主复制
  • 多harbor实例共享后端存储

1.1 方案说明

1. 双主复制

所谓的双主复制 其实就是复用主从同步实现两个harbor节点之间的双向同步,来保证数据的一致性,然后在两台harbor前端顶一个负载均衡器将进来的请求分流到不同的实例中去,只要有一个实例中有了新的镜像,就是自动的同步复制到另外的的实例中去,这样实现了负载均衡,也避免了单点故障,在一定程度上实现了Harbor的高可用性:

这个方案有一个问题就是有可能两个Harbor实例中的数据不一致。假设如果一个实例A挂掉了,这个时候有新的镜像进来,那么新的镜像就会在另外一个实例B中,后面即使恢复了挂掉的A实例,Harbor实例B也不会自动去同步镜像,这样只能手动的先关掉Harbor实例B的复制策略,然后再开启复制策略,才能让实例B数据同步,让两个实例的数据一致。

在实际生产使用中,主从复制十分的不靠谱,所以这里就不配置了。

2. 多harbor实例共享后端存储

利用共享存储和共享数据库来实现服务的高可用性和数据的冗余

这个方案在实际生产环境中部署需要考虑三个问题:

  • 共享存储的选取,Harbor的后端存储目前支持AWS S3、Openstack Swift, Ceph等,在我们的实验环境里,就直接使用nfs。
  • Session在不同的实例上共享,这个现在其实已经不是问题了,在最新的harbor中,默认session会存放在redis中,我们只需要将redis独立出来即可。可以通过redis sentinel或者redis cluster等方式来保证redis的可用性。在我们的实验环境里,仍然使用单台redis。
  • Harbor多实例数据库问题,这个也只需要将harbor中的数据库拆出来独立部署即可。让多实例共用一个外部数据库,数据库的高可用也可以通过数据库的高可用方案保证。可选择的数据库包括PostgreSql,mysql等等。

1.2 部署高可用(多harbor实例共享后端存储)

1. 服务器划分

服务器 IP 说明
k8s-harbor-01.xx.net 192.168.17.220 harbor1服务器
k8s-harbor-02.xx.net 192.168.17.221 harbor2服务器
k8s-harbor-lb-01.xx.net 192.168.17.225 nginx,redis,mysql,nfs

我们将在k8s-harbor-lb-01.xx.net部署nginx,redis,mysql,nfs等服务,生产环境中应该分开,并且配置成为高可用

2. 安装harbor(先部署一套Harbor,用于将其所有表结构导出)

下载安装包并上传到服务器

bash 复制代码
tar xvf harbor-offline-installer-v2.7.2.tgz
cd harbor

mkdir certs	#证书
cd certs/
openssl genrsa -out ./harbor-ca.key		#key
openssl req -x509 -new -nodes -key ./harbor-ca.key -subj "/CN=harbor.xx.net" -days 7120 -out ./harbor-ca.crt	#认证

配置harbor.yml

bash 复制代码
cp harbor.yml.tmpl harbor.yml
[root@k8s-harbor-01 harbor]# egrep -v '^$|^#|^  #' harbor.yml
hostname: harbor.xx.net
http:
  port: 80
https:
  port: 443
  certificate: /opt/harbor/certs/harbor-ca.crt
  private_key: /opt/harbor/certs/harbor-ca.key
harbor_admin_password: 123456
...

启动harbor

bash 复制代码
./install.sh --with-trivy --with-chartmuseum
--with-trivy #镜像漏洞检测
--with-chartmuseum #Chart仓库服务

本节搭建的目的是导出postgresql数据库到其他服务器,接着导出数据库

bash 复制代码
docker ps
docker exec -it harbor-db /bin/bash

进入容器

bash 复制代码
## 执行 psql 进入数据库
postgres [ / ]$ psql
psql (9.6.14)
Type "help" for help.

## 查看当前所有的数据库,postgres、template0、template1为默认数据库
postgres=# \l
                                   List of databases
     Name     |  Owner   | Encoding |   Collate   |    Ctype    |   Access privileges   
--------------+----------+----------+-------------+-------------+-----------------------
 notaryserver | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 | =Tc/postgres         +
              |          |          |             |             | postgres=CTc/postgres+
              |          |          |             |             | server=CTc/postgres
 notarysigner | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 | =Tc/postgres         +
              |          |          |             |             | postgres=CTc/postgres+
              |          |          |             |             | signer=CTc/postgres
 postgres     | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 | 
 registry     | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 | 
 template0    | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 | =c/postgres          +
              |          |          |             |             | postgres=CTc/postgres
 template1    | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 | =c/postgres          +
              |          |          |             |             | postgres=CTc/postgres
(6 rows)

postgres=# \q
## 导出表结构及数据
postgres [ / ]$ pg_dump -U postgres registry > /tmp/registry.sql
postgres [ / ]$ pg_dump -U postgres notaryserver > /tmp/notaryserver.sql
postgres [ / ]$ pg_dump -U postgres notarysigner > /tmp/notarysigner.sql
    -U 数据库用户
    -p 访问端口
    -f 指定文件,和 > 功能一样
    -h 指定数据库地址
    -s 表示只导出表结构,不导数据

导出到宿主机

bash 复制代码
docker cp 8d69069a2cd7:/tmp/registry.sql ./
docker cp 8d69069a2cd7:/tmp/notaryserver.sql ./
docker cp 8d69069a2cd7:/tmp/notarysigner.sql ./

3. 安装Postgresql

bash 复制代码
yum install -y https://download.postgresql.org/pub/repos/yum/reporpms/EL-7-x86_64/pgdg-redhat-repo-latest.noarch.rpm
yum install -y postgresql13-server

/usr/pgsql-13/bin/postgresql-13-setup initdb #初始化数据库

# 修改远程访问配置
vim /var/lib/pgsql/13/data/postgresql.conf
...
#将 listen_addresses = 'localhost' 修改为
listen_addresses = '*'

# 添加信任的远程连接,生产中不要添加0.0.0.0
vim /var/lib/pgsql/13/data/pg_hba.conf
...
host    all             all             0.0.0.0/0               trust
# host    all             all             0.0.0.0/0               md5
# 最后一列如果是trust,则登录pg不需要密码,若为md5,则需要密码

# start and enable server
systemctl enable postgresql-13
systemctl start postgresql-13

# 检查服务是否启动成功
#ps看进程 或 ss看端口号

#给postgresql设置密码,增强安全性
su - postgres
-bash-4.2$ psql
## 直接写入新密码
postgres=# \password
Enter new password for user "postgres":
Enter it again:
postgres=# \q

#验证
psql -U postgres -h localhost
Password for user postgres:
psql (13.14)
Type "help" for help.

postgres=# exit

#把全部信任改为指定IP
[root@k8s-harbor-lb-01 ~]# tail -3 /var/lib/pgsql/13/data/pg_hba.conf
host    all             all             192.168.17.220/24               trust
host    all             all             192.168.17.221/24               trust
#host   all             all             0.0.0.0/0               trust

将备份的数据,导入进单独部署的postgresql中

bash 复制代码
## 创建数据库
postgres=# CREATE DATABASE registry;
postgres=# CREATE DATABASE notaryserver;
postgres=# CREATE DATABASE notarysigner;

将harbor服务器的导出的SQL拷贝到本机

bash 复制代码
scp -r 192.168.17.220:/opt/harbor/postgresql_export /opt/postgresql_export
notaryserver.sql                                                                                        100%  491   213.2KB/s   00:00
notarysigner.sql                                                                                        100%  491   281.3KB/s   00:00
registry.sql                                                                                            100%  101KB  19.7MB/s   00:00

导入数据

bash 复制代码
## 
psql -h localhost -U postgres -p 5432 -d registry -f registry.sql 
psql -h localhost -U postgres -p 5432 -d notaryserver -f notaryserver.sql 
psql -h localhost -U postgres -p 5432 -d notarysigner -f notarysigner.sql 
    -U 数据库用户
    -p 访问端口
    -f 指定文件,和 < 功能一样
    -h 指定数据库地址
    -d 指定数据库名

4. 安装nginx

bash 复制代码
rpm -Uvh http://nginx.org/packages/centos/7/noarch/RPMS/nginx-release-centos-7-0.el7.ngx.noarch.rpm
yum install -y nginx

配置nginx.conf

bash 复制代码
[root@k8s-harbor-lb-01 ~]# cat /etc/nginx/nginx.conf
...
    include /etc/nginx/conf.d/*.conf;

    upstream harborsvr {
        server 192.168.17.220:80 weight=2;
        server 192.168.17.221:80 weight=1;
    }

    server {
        listen 80;
        server_name 192.168.17.225;
        location / {
                proxy_pass http://harborsvr;

                }
        }
}

启动

bash 复制代码
systemctl start nginx
systemctl status nginx
systemctl enable nginx

5. 安装nfs

bash 复制代码
yum install -y nfs-utils

# 编辑/etc/exports文件
/data *(rw,no_root_squash)

chmod 777 -R /data

systemctl start nfs-server
systemctl enable nfs-server

mdkir /data/
mount -t nfs `hostname`:/data/ /data/

6. 安装redis

bash 复制代码
yum install epel-release -y
yum install redis -y

## 
vim /etc/redis.conf
...
bind 0.0.0.0 # 设置所有主机可以连接
requirepass 123456 # 设置客户端连接密码
daemonize yes # 打开守护进程模式
...

## 启动redis
systemctl start redis
systemctl enable redis

7. 部署harbor

我们将第2段中,部署的harbor进行修改配置文件

bash 复制代码
[root@k8s-harbor-01 harbor]# docker-compose down

编辑配置文件,需要更改的主要有以下几点:

1.hostname 改为主机ip或完全限定域名,不要使用127.0.0.1或localhost

2.https选项,如需要,指定crt和key的路径,若不需要,直接注释掉

3.harbor_admin_password,默认密码,可以更改

4.data_volume,数据默认存储位置,设计为共享路径

5.注释掉database模块 及 Clair模块

6.开启external_database 和 external_redis模块及正确配置其中参数

7.集群内所有harbor配置均一样,改一下hostname值即可

修改配置文件(经过后面挂掉之后,重新改的配置文件)

bash 复制代码
[root@k8s-harbor-01 harbor]# egrep -v '^$|^#|^  #' harbor.yml
hostname: harbor.xx.net
http:
  port: 80
https:
  port: 443
  certificate: /opt/harbor/certs/harbor-ca.crt
  private_key: /opt/harbor/certs/harbor-ca.key
harbor_admin_password: 123456
data_volume: /data
trivy:
  ignore_unfixed: false
  skip_update: false
  offline_scan: false
  security_check: vuln
  insecure: false
jobservice:
  max_job_workers: 10
notification:
  webhook_job_max_retry: 10
chart:
  absolute_url: disabled
log:
  level: info
  local:
    # Log files are rotated log_rotate_count times before being removed. If count is 0, old versions are removed rather than rotated.
    rotate_count: 50
    # Log files are rotated only if they grow bigger than log_rotate_size bytes. If size is followed by k, the size is assumed to be in kilobytes.
    # If the M is used, the size is in megabytes, and if G is used, the size is in gigabytes. So size 100, size 100k, size 100M and size 100G
    # are all valid.
    rotate_size: 200M
    # The directory on your host that store log
    location: /var/log/harbor
_version: 2.7.0
external_database:
  harbor:
    host: 192.168.17.225
    port: 5432
    db_name: registry
    username: postgres
    password: 123456
    ssl_mode: disable
    max_idle_conns: 2
    max_open_conns: 0
  notary_signer:
    host: 192.168.17.225
    port: 5432
    db_name: notarysigner
    username: postgres
    password: 123456
    ssl_mode: disable
  notary_server:
    host: 192.168.17.225
    port: 5432
    db_name: notaryserver
    username: postgres
    password: 123456
    ssl_mode: disable
external_redis:
   host: 192.168.17.225:6379
   password: 123456
   registry_db_index: 1
   jobservice_db_index: 2
   chartmuseum_db_index: 3
   chair_db_index: 4
   trivy_db_index: 5
   idle_timeout_seconds: 30
proxy:
  http_proxy:
  https_proxy:
  no_proxy:
  components:
    - core
    - jobservice
    - trivy
metric:
  enabled: false
  port: 9090
  path: /metrics
upload_purging:
  enabled: true
  age: 168h
  interval: 24h
  dryrun: false
cache:
  enabled: false
  expire_hours: 24

启动harbor的过程中发现jobservice容器无法启动,怀疑是数据库连接失败

bash 复制代码
[root@k8s-harbor-01 harbor]# docker ps
CONTAINER ID   IMAGE                                  COMMAND                  CREATED        STATUS                             PORTS                                                                            NAMES
767ac8431315   goharbor/harbor-jobservice:v2.7.2      "/harbor/entrypoint...."   18 hours ago   Restarting (2) 17 seconds ago                                                                                       harbor-jobservice
...

日志

bash 复制代码
[root@k8s-harbor-01 harbor]# docker logs 767
Appending internal tls trust CA to ca-bundle ...
find: '/etc/harbor/ssl': No such file or directory
Internal tls trust CA appending is Done.
2024-02-21T17:00:15Z [ERROR] [/pkg/registry/client.go:82]: Failed to parse REGISTRY_HTTP_CLIENT_TIMEOUT: strconv.ParseInt: parsing "": invalid syntax, use default value: 30m0s
2024-02-21T17:00:15Z [INFO] [/controller/artifact/annotation/parser.go:71]: the annotation parser to parser artifact annotation version v1alpha1 registered
2024-02-21T17:00:15Z [INFO] [/controller/artifact/processor/processor.go:59]: the processor to process media type application/vnd.wasm.config.v1+json registered
2024-02-21T17:00:15Z [ERROR] [/lib/config/config.go:81]: failed to get config manager
2024-02-21T17:00:15Z [ERROR] [/lib/config/config.go:81]: failed to get config manager

在harbor服务器安装postgresql客户端,

bash 复制代码
yum install https://download.postgresql.org/pub/repos/yum/reporpms/EL-7-x86_64/pgdg-redhat-repo-latest.noarch.rpm -y
yum install postgresql13 -y

[root@k8s-harbor-01 harbor]# psql -U postgres -h 192.168.17.225 -p 5432
psql (13.14)
Type "help" for help.

postgres=# \l
                                   List of databases
     Name     |  Owner   | Encoding |   Collate   |    Ctype    |   Access privileges
--------------+----------+----------+-------------+-------------+-----------------------
 notaryserver | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 |
 notarysigner | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 |
 postgres     | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 |
 registry     | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 |
 template0    | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 | =c/postgres          +
              |          |          |             |             | postgres=CTc/postgres
 template1    | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 | =c/postgres          +
              |          |          |             |             | postgres=CTc/postgres
(6 rows)

postgres=#

发现可以连接,注释掉缓存数据库redis配置,发现可以正常启动,说明是redis配置有问题

bash 复制代码
[root@k8s-harbor-01 harbor]# docker ps
CONTAINER ID   IMAGE                                  COMMAND                  CREATED              STATUS                        PORTS                                                                            NAMES
dccc714a7abe   goharbor/harbor-jobservice:v2.7.2      "/harbor/entrypoint...."   About a minute ago   Up About a minute (healthy)                                                                                    harbor-jobservice

经过调查,原来是redis的password修改过,和默认的不匹配
harbor1服务器配置完成,接下来配置harbor2服务器,相同配置,改下域名就可以了

bash 复制代码
[root@k8s-harbor-02 harbor]# grep hostname harbor.yml
# The IP address or hostname to access admin UI and registry service.
hostname: harbor2.xx.net

8. 修改nginx配置

由于我们安装的harbor通过http跳转到https访问,所以前面设置的nginx的负载均衡的配置需要进行修改,否则无法访问

bash 复制代码
[root@k8s-harbor-lb-01 nginx]# cat /etc/nginx/nginx.conf
...
http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    sendfile        on;
    #tcp_nopush     on;

    keepalive_timeout  65;

    #gzip  on;

    include /etc/nginx/conf.d/*.conf;

    upstream harborsvrs {
        server 192.168.17.220:443 weight=2;
        server 192.168.17.221:443 weight=1;
    }

    server {
        listen 443;
        server_name 192.168.17.225;

        location / {
                proxy_pass https://harborsvrs/;
                proxy_set_header   Host $host;
                proxy_set_header   X-Real-IP $remote_addr;
                proxy_set_header   X-Forwarded-Proto $scheme;

               }
        }
}

修改后,重启nginx

9. docker登录harbor

bash 复制代码
[root@k8s-master-01 ansible]# docker login 192.168.17.225:443
Username: admin
Password:
Error response from daemon: Get "http://192.168.17.225:443/v2/": Get "https://harbor1.xx.net/service/token?account=admin&client_id=docker&offline_token=true&service=harbor-registry": dial tcp: lookup harbor1.xx.net on 8.8.8.8:53: no such host

解析不了harbor的域名,在没有DNS服务器的情况下,修改/etc/hosts文件

bash 复制代码
[root@k8s-master-01 ansible]# cat /etc/hosts
...
192.168.17.220 k8s-harbor-01.xx.net harbor1 harbor1.xx.net
192.168.17.221 k8s-harbor-02.xx.net harbor2 harbor2.xx.net

登录

bash 复制代码
[root@k8s-master-01 ansible]# docker login 192.168.17.225:443
Username: admin
Password:
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store

Login Succeeded

由于推送的时候出现问题,笔者一直未能解决,所以将harbor回退到使用http协议

bash 复制代码
[root@k8s-master-01 ansible]# docker push harbor2.xx.net/alpine/alpine:latest
The push refers to repository [harbor2.xx.net/alpine/alpine]
d4fc045c9e3a: Layer already exists
unauthorized: unauthorized to access repository: alpine/alpine, action: push: unauthorized to access repository: alpine/alpine, action: push

10. harbor修改

bash 复制代码
[root@k8s-harbor-01 harbor]# head -20 harbor.yml
# Configuration file of Harbor

# The IP address or hostname to access admin UI and registry service.
# DO NOT use localhost or 127.0.0.1, because Harbor needs to be accessed by external clients.
#hostname: harbor1.xx.net
hostname: 192.168.17.220

# http related config
http:
  # port for http, default is 80. If https enabled, this port will redirect to https port
  port: 80

# https related config
#https:
  # https port for harbor, default is 443
#  port: 443
  # The path of cert and key files for nginx
#  certificate: /opt/harbor/certs/harbor-ca.crt
#  private_key: /opt/harbor/certs/harbor-ca.key

修改harbor1和harbor2服务器的配置文件,并重启harbor

11. 修改nginx

bash 复制代码
http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    sendfile        on;
    #tcp_nopush     on;

    keepalive_timeout  65;

    #gzip  on;

    include /etc/nginx/conf.d/*.conf;

    upstream harborsvrs {
        server 192.168.17.220:80 weight=2;
        server 192.168.17.221:80 weight=1;
    }

    server {
        listen 80;
        server_name 192.168.17.225;
        autoindex       on;
        location / {
                proxy_pass http://harborsvrs/;

                }
        }
}

通过负载均衡可以访问harbor

12. docker推送

bash 复制代码
[root@k8s-master-01 ansible]# docker login 192.168.17.221:80 -u admin
Password:
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store

Login Succeeded
[root@k8s-master-01 ansible]# docker tag alpine:latest 192.168.17.221:80/alpine/alpine:v1
[root@k8s-master-01 ansible]# docker push 192.168.17.221:80/alpine/alpine:v1
The push refers to repository [192.168.17.221:80/alpine/alpine]
d4fc045c9e3a: Layer already exists
v1: digest: sha256:6457d53fb065d6f250e1504b9bc42d5b6c65941d57532c072d929dd0628977d0 size: 528
[root@k8s-master-01 ansible]# docker login 192.168.17.220:80 -u admin
Password:
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store

Login Succeeded
[root@k8s-master-01 ansible]# docker push 192.168.17.220:80/alpine/alpine:latest
The push refers to repository [192.168.17.220:80/alpine/alpine]
d4fc045c9e3a: Layer already exists
latest: digest: sha256:6457d53fb065d6f250e1504b9bc42d5b6c65941d57532c072d929dd0628977d0 size: 528

在其他服务器登录harbor后,也可以正常推送

相关推荐
数巨小码人1 小时前
Linux常用指令
linux·运维·服务器
Linux运维老纪2 小时前
Linux之七大难命令(The Seven Difficult Commands of Linux)
linux·运维·服务器·云计算·运维开发
张书名2 小时前
高性能服务器配置经验指南3——安装服务器可能遇到的问题及解决方法
运维·服务器
李菠菜3 小时前
Kubernetes上通过Helm部署高可用Redis集群
docker·容器·kubernetes
李菠菜3 小时前
修改KubeSphere外网访问端口
docker·容器·kubernetes
前进的程序员3 小时前
在Linux驱动开发中使用DeepSeek的方法
linux·运维·服务器·人工智能
福大大架构师每日一题3 小时前
docker v28.1.1 正式发布!修复关键Bug,网络与安全性再升级
网络·docker·bug
Tome--3 小时前
运维面试题01
运维
银河麒麟操作系统3 小时前
【银河麒麟高级服务器操作系统】磁盘只读问题分析
java·linux·运维·服务器·jvm
一个小坑货4 小时前
Docker 部署 PostgreSQL 数据库
数据库·docker·postgresql