ClickHouse 5节点集群安装

ClickHouse 5节点集群安装

在此架构中，配置了五台服务器。其中两个用于托管数据副本。其他三台服务器用于协调数据的复制。在此示例中，我们将创建一个数据库和表，将使用 ReplicatedMergeTree 表引擎在两个数据节点之间复制该数据库和表。

官方文档：https://clickhouse.com/docs/en/architecture/replication

部署环境

节点清单：

主机名	节点IP	操作系统	节点配置	描述
clickhouse-01	192.168.72.51	Ubuntu22.04	2C/4G/100G DISK	Clickhose server, client
clickhouse-02	192.168.72.52	Ubuntu22.04	2C/4G/100G DISK	Clickhose server, client
clickhouse-keeper-01	192.168.72.53	Ubuntu22.04	2C/4G/100G DISK	Clikhouse keeper
clickhouse-keeper-02	192.168.72.54	Ubuntu22.04	2C/4G/100G DISK	Clikhouse keeper
clickhouse-keeper-03	192.168.72.55	Ubuntu22.04	2C/4G/100G DISK	Clikhouse keeper

说明：

在生产环境中，我们强烈建议为 ClickHouse keeper 使用专用主机。在测试环境中，可以在同一服务器上组合运行 ClickHouse Server 和 ClickHouse Keeper。另一个基本示例"横向扩展"就使用了这种方法。在此示例中，我们介绍了将 Keeper 与 ClickHouse Server 分离的推荐方法。 Keeper 服务器可以更小，4GB RAM 通常足以用于每个 Keeper 服务器，直到您的 ClickHouse 服务器变得非常大。

在所有节点上配置主机名

hostnamectl set-hostname clickhouse-01
hostnamectl set-hostname clickhouse-02
hostnamectl set-hostname clickhouse-keeper-01
hostnamectl set-hostname clickhouse-keeper-02
hostnamectl set-hostname clickhouse-keeper-03

在所有节点上编辑 /etc/hosts 文件

cat >/etc/hosts<<EOF
192.168.72.51 clickhouse-01 clickhouse-01.example.com
192.168.72.52 clickhouse-02 clickhouse-02.example.com
192.168.72.53 clickhouse-keeper-01 clickhouse-keeper-01.example.com
192.168.72.54 clickhouse-keeper-02 clickhouse-keeper-02.example.com
192.168.72.55 clickhouse-keeper-03 clickhouse-keeper-03.example.com
EOF

安装clickhouse

clickhouse-01和clickhouse-02节点执行

在clickhouse-01和clickhouse-02节点上需要安装ClickHouse-server及client

sudo apt-get install -y apt-transport-https ca-certificates dirmngr
sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 8919F6BD2B48D754
echo "deb https://packages.clickhouse.com/deb stable main" | sudo tee \
    /etc/apt/sources.list.d/clickhouse.list
sudo apt-get update
sudo apt-get install -y clickhouse-server clickhouse-client

clickhouse-keeper-01~03节点执行

在clickhouse-keeper-01~03节点上仅安装clickhose-keeper 。

sudo apt-get install -y apt-transport-https ca-certificates dirmngr
sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 8919F6BD2B48D754
echo "deb https://packages.clickhouse.com/deb stable main" | sudo tee \
    /etc/apt/sources.list.d/clickhouse.list
sudo apt-get update
sudo apt-get install -y clickhouse-keeper

创建clickhose-keeper相关目录

mkdir -p /etc/clickhouse-keeper/config.d
mkdir -p /var/log/clickhouse-keeper
mkdir -p /var/lib/clickhouse-keeper/coordination/log
mkdir -p /var/lib/clickhouse-keeper/coordination/snapshots
mkdir -p /var/lib/clickhouse-keeper/cores
chown -R clickhouse.clickhouse /etc/clickhouse-keeper /var/log/clickhouse-keeper /var/lib/clickhouse-keeper

clickhouse-01配置

对于 clickhouse-01 有五个配置文件。您可以选择将这些文件合并为一个文件，但为了文档的清晰性，单独查看它们可能会更简单。当您通读配置文件时，您会发现 clickhouse-01 和 clickhouse-02 之间的大部分配置是相同的；差异将被突出显示。

网络和日志记录配置

这些值可以根据您的意愿进行定制。此示例配置为您提供：

• 调试日志将以 1000M 滚动 3 次
• 使用clickhouse-client连接时显示的名称是cluster_1S_2R node 1
• ClickHouse 将侦听 IPV4 网络的端口 8123 和 9000。

clickhouse-01 上的 /etc/clickhouse-server/config.d/network-and-logging.xml

root@clickhouse-01:~# cat /etc/clickhouse-server/config.d/network-and-logging.xml
<clickhouse>
    <logger>
        <level>debug</level>
        <log>/var/log/clickhouse-server/clickhouse-server.log</log>
        <errorlog>/var/log/clickhouse-server/clickhouse-server.err.log</errorlog>
        <size>1000M</size>
        <count>3</count>
    </logger>
    <display_name>cluster_1S_2R node 1</display_name>
    <listen_host>0.0.0.0</listen_host>
    <http_port>8123</http_port>
    <tcp_port>9000</tcp_port>
</clickhouse>

宏配置

宏shard和replica降低了分布式 DDL 的复杂性。配置的值会自动替换到您的 DDL 查询中，从而简化您的 DDL。此配置的宏指定每个节点的分片和副本数量。

在此 1 分片 2 副本示例中，副本宏是 clickhouse-01 上的replica_1和 clickhouse-02 上的replica_2 。 clickhouse-01 和 clickhouse-02 上的分片宏均为1因为只有一个分片。

clickhouse-01 上的 /etc/clickhouse-server/config.d/macros.xml

root@clickhouse-01:~# cat /etc/clickhouse-server/config.d/macros.xml
<clickhouse>
    <macros>
        <shard>01</shard>
        <replica>01</replica>
        <cluster>cluster_1S_2R</cluster>
    </macros>
</clickhouse>

复制和分片配置

从顶部开始：

• XML 的remote_servers 部分指定环境中的每个集群。属性replace=true将默认ClickHouse配置中的示例remote_servers替换为此文件中指定的remote_server配置。如果没有此属性，此文件中的远程服务器将被附加到默认的示例列表中。
• 在此示例中，有一个名为cluster_1S_2R的集群。
• 为名为cluster_1S_2R的集群创建一个机密，其值为mysecretphrase 。该秘密在环境中的所有远程服务器之间共享，以确保正确的服务器连接在一起。
• 集群cluster_1S_2R有 1 个分片和 2 个副本。查看本文档开头的架构图，并将其与下面 XML 中的shard定义进行比较。分片定义包含两个副本。指定每个副本的主机和端口。一个副本存储在clickhouse-01上，另一个副本存储在clickhouse-02上。
• 分片的内部复制设置为 true。每个分片都可以在配置文件中定义internal_replication 参数。如果该参数设置为true，则写操作会选择第一个健康的副本并向其写入数据。

clickhouse-01 上的 /etc/clickhouse-server/config.d/remote-servers.xml

root@clickhouse-01:~# cat /etc/clickhouse-server/config.d/remote-servers.xml
<clickhouse>
    <remote_servers replace="true">
        <cluster_1S_2R>
            <secret>mysecretphrase</secret>
            <shard>
                <internal_replication>true</internal_replication>
                <replica>
                    <host>clickhouse-01</host>
                    <port>9000</port>
                </replica>
                <replica>
                    <host>clickhouse-02</host>
                    <port>9000</port>
                </replica>
            </shard>
        </cluster_1S_2R>
    </remote_servers>
</clickhouse>

配置Keeper的使用

此配置文件use-keeper.xml将 ClickHouse Server 配置为使用 ClickHouse Keeper 来协调复制和分布式 DDL。此文件指定 ClickHouse Server 应在端口 9181 上的节点 clickhouse-keeper-01 - 03 上使用 Keeper，并且该文件在clickhouse-01和clickhouse-02上相同。

clickhouse-01 上的 /etc/clickhouse-server/config.d/use-keeper.xml

root@clickhouse-01:~# cat /etc/clickhouse-server/config.d/use-keeper.xml
<clickhouse>
    <zookeeper>
        <!-- where are the ZK nodes -->
        <node>
            <host>clickhouse-keeper-01</host>
            <port>9181</port>
        </node>
        <node>
            <host>clickhouse-keeper-02</host>
            <port>9181</port>
        </node>
        <node>
            <host>clickhouse-keeper-03</host>
            <port>9181</port>
        </node>
    </zookeeper>
</clickhouse>

clickhouse-02配置

由于 clickhouse-01 和 clickhouse-02 上的配置非常相似，这里仅指出差异。

网络和日志记录配置

该文件在 clickhouse-01 和 clickhouse-02 上都是相同的，但display_name除外。

clickhouse-02 上的 /etc/clickhouse-server/config.d/network-and-logging.xml

root@clickhouse-02:~# cat /etc/clickhouse-server/config.d/network-and-logging.xml
<clickhouse>
    <logger>
        <level>debug</level>
        <log>/var/log/clickhouse-server/clickhouse-server.log</log>
        <errorlog>/var/log/clickhouse-server/clickhouse-server.err.log</errorlog>
        <size>1000M</size>
        <count>3</count>
    </logger>
    <display_name>cluster_1S_2R node 2</display_name>
    <listen_host>0.0.0.0</listen_host>
    <http_port>8123</http_port>
    <tcp_port>9000</tcp_port>
</clickhouse>

宏配置

clickhouse-01 和 clickhouse-02 之间的宏配置不同。 replica在此节点上设置为02 。

clickhouse-02 上的 /etc/clickhouse-server/config.d/macros.xml

root@clickhouse-02:~# cat /etc/clickhouse-server/config.d/macros.xml
<clickhouse>
    <macros>
        <shard>01</shard>
        <replica>02</replica>
        <cluster>cluster_1S_2R</cluster>
    </macros>
</clickhouse>

复制和分片配置

该文件在 clickhouse-01 和 clickhouse-02 上是相同的。

clickhouse-02 上的 /etc/clickhouse-server/config.d/remote-servers.xml

root@clickhouse-02:~# cat /etc/clickhouse-server/config.d/remote-servers.xml
<clickhouse>
    <remote_servers replace="true">
        <cluster_1S_2R>
            <secret>mysecretphrase</secret>
            <shard>
                <internal_replication>true</internal_replication>
                <replica>
                    <host>clickhouse-01</host>
                    <port>9000</port>
                </replica>
                <replica>
                    <host>clickhouse-02</host>
                    <port>9000</port>
                </replica>
            </shard>
        </cluster_1S_2R>
    </remote_servers>
</clickhouse>

配置Keeper的使用

该文件在 clickhouse-01 和 clickhouse-02 上是相同的。

clickhouse-02 上的 /etc/clickhouse-server/config.d/use-keeper.xml

root@clickhouse-02:~# cat /etc/clickhouse-server/config.d/use-keeper.xml
<clickhouse>
    <zookeeper>
        <!-- where are the ZK nodes -->
        <node>
            <host>clickhouse-keeper-01</host>
            <port>9181</port>
        </node>
        <node>
            <host>clickhouse-keeper-02</host>
            <port>9181</port>
        </node>
        <node>
            <host>clickhouse-keeper-03</host>
            <port>9181</port>
        </node>
    </zookeeper>
</clickhouse>

clickhouse-keeper-01配置

最佳实践

通过编辑配置文件来配置 ClickHouse Keeper 时，您应该：

• 备份 /etc/clickhouse-keeper/keeper_config.xml
• 编辑 /etc/clickhouse-keeper/keeper_config.xml 文件

ClickHouse Keeper 提供数据复制和分布式 DDL 查询执行的协调系统。 ClickHouse Keeper 与 Apache ZooKeeper 兼容。此配置在端口 9181 上启用 ClickHouse Keeper。突出显示的行指定该 Keeper 实例的 server_id 为 1。这是三台服务器的enable-keeper.xml文件中的唯一区别。 clickhouse-keeper-02将server_id设置为2 ， clickhouse-keeper-03将server_id设置为3 。 raft 配置部分在所有三台服务器上都是相同的，下面突出显示以向您展示 raft 配置中server_id和server实例之间的关系。

说明

如果出于任何原因更换或重建 Keeper 节点，请勿重复使用现有的server_id 。例如，如果重建了server_id为2的Keeper节点，则将其server_id设置为4或更高。

备份所有节点keeper_config.xml配置

# 备份配置
cp /etc/clickhouse-keeper/keeper_config.xml{,.bak}
# 清空默认配置
echo > /etc/clickhouse-keeper/keeper_config.xml

clickhouse-keeper-01 上的 /etc/clickhouse-keeper/keeper_config.xml

root@clickhouse-keeper-01:~# cat /etc/clickhouse-keeper/keeper_config.xml
<clickhouse>
    <logger>
        <level>trace</level>
        <log>/var/log/clickhouse-keeper/clickhouse-keeper.log</log>
        <errorlog>/var/log/clickhouse-keeper/clickhouse-keeper.err.log</errorlog>
        <size>1000M</size>
        <count>3</count>
    </logger>
    <listen_host>0.0.0.0</listen_host>
    <keeper_server>
        <tcp_port>9181</tcp_port>
        <server_id>1</server_id>
        <log_storage_path>/var/lib/clickhouse/coordination/log</log_storage_path>
        <snapshot_storage_path>/var/lib/clickhouse/coordination/snapshots</snapshot_storage_path>
        <coordination_settings>
            <operation_timeout_ms>10000</operation_timeout_ms>
            <session_timeout_ms>30000</session_timeout_ms>
            <raft_logs_level>trace</raft_logs_level>
        </coordination_settings>
        <raft_configuration>
            <server>
                <id>1</id>
                <hostname>clickhouse-keeper-01</hostname>
                <port>9234</port>
            </server>
            <server>
                <id>2</id>
                <hostname>clickhouse-keeper-02</hostname>
                <port>9234</port>
            </server>
            <server>
                <id>3</id>
                <hostname>clickhouse-keeper-03</hostname>
                <port>9234</port>
            </server>
        </raft_configuration>
    </keeper_server>
</clickhouse>

clickhouse-keeper-02配置

clickhouse-keeper-01和clickhouse-keeper-02之间只有一行差异。该节点上的server_id设置为2 。

clickhouse-keeper-02 上的 /etc/clickhouse-keeper/keeper_config.xml

root@clickhouse-keeper-02:~# cat /etc/clickhouse-keeper/keeper_config.xml
<clickhouse>
    <logger>
        <level>trace</level>
        <log>/var/log/clickhouse-keeper/clickhouse-keeper.log</log>
        <errorlog>/var/log/clickhouse-keeper/clickhouse-keeper.err.log</errorlog>
        <size>1000M</size>
        <count>3</count>
    </logger>
    <listen_host>0.0.0.0</listen_host>
    <keeper_server>
        <tcp_port>9181</tcp_port>
        <server_id>2</server_id>
        <log_storage_path>/var/lib/clickhouse/coordination/log</log_storage_path>
        <snapshot_storage_path>/var/lib/clickhouse/coordination/snapshots</snapshot_storage_path>
        <coordination_settings>
            <operation_timeout_ms>10000</operation_timeout_ms>
            <session_timeout_ms>30000</session_timeout_ms>
            <raft_logs_level>trace</raft_logs_level>
        </coordination_settings>
        <raft_configuration>
            <server>
                <id>1</id>
                <hostname>clickhouse-keeper-01</hostname>
                <port>9234</port>
            </server>
            <server>
                <id>2</id>
                <hostname>clickhouse-keeper-02</hostname>
                <port>9234</port>
            </server>
            <server>
                <id>3</id>
                <hostname>clickhouse-keeper-03</hostname>
                <port>9234</port>
            </server>
        </raft_configuration>
    </keeper_server>
</clickhouse>

clickhouse-keeper-03配置

clickhouse-keeper-01和clickhouse-keeper-03之间只有一行差异。该节点上的server_id设置为3 。

clickhouse-keeper-03 上的 /etc/clickhouse-keeper/keeper_config.xml

root@clickhouse-keeper-03:~# cat /etc/clickhouse-keeper/keeper_config.xml
<clickhouse>
    <logger>
        <level>trace</level>
        <log>/var/log/clickhouse-keeper/clickhouse-keeper.log</log>
        <errorlog>/var/log/clickhouse-keeper/clickhouse-keeper.err.log</errorlog>
        <size>1000M</size>
        <count>3</count>
    </logger>
    <listen_host>0.0.0.0</listen_host>
    <keeper_server>
        <tcp_port>9181</tcp_port>
        <server_id>3</server_id>
        <log_storage_path>/var/lib/clickhouse/coordination/log</log_storage_path>
        <snapshot_storage_path>/var/lib/clickhouse/coordination/snapshots</snapshot_storage_path>
        <coordination_settings>
            <operation_timeout_ms>10000</operation_timeout_ms>
            <session_timeout_ms>30000</session_timeout_ms>
            <raft_logs_level>trace</raft_logs_level>
        </coordination_settings>
        <raft_configuration>
            <server>
                <id>1</id>
                <hostname>clickhouse-keeper-01</hostname>
                <port>9234</port>
            </server>
            <server>
                <id>2</id>
                <hostname>clickhouse-keeper-02</hostname>
                <port>9234</port>
            </server>
            <server>
                <id>3</id>
                <hostname>clickhouse-keeper-03</hostname>
                <port>9234</port>
            </server>
        </raft_configuration>
    </keeper_server>
</clickhouse>

启动服务

在clickhouse-keeper-01~03节点执行；

启动服务

systemctl enable --now clickhouse-keeper.service

确认clickhouse-keeper-01服务运行状态

root@clickhouse-keeper-01:~# systemctl status clickhouse-keeper.service 
● clickhouse-keeper.service - ClickHouse Keeper - zookeeper compatible distributed coordination server
     Loaded: loaded (/lib/systemd/system/clickhouse-keeper.service; enabled; vendor preset: enabled)
     Active: active (running) since Sun 2024-10-27 19:44:26 CST; 3h 0min ago
   Main PID: 3460 (clickhouse-keep)
      Tasks: 41 (limit: 4556)
     Memory: 58.8M
        CPU: 1min 13.000s
     CGroup: /system.slice/clickhouse-keeper.service
             └─3460 /usr/bin/clickhouse-keeper --config=/etc/clickhouse-keeper/keeper_config.xml --pid-file=/run/clickhouse-keeper/clickhouse-keeper.pid

Oct 27 19:44:26 clickhouse-keeper-01 systemd[1]: Started ClickHouse Keeper - zookeeper compatible distributed coordination server.
Oct 27 19:44:26 clickhouse-keeper-01 clickhouse-keeper[3460]: Processing configuration file '/etc/clickhouse-keeper/keeper_config.xml'.
Oct 27 19:44:26 clickhouse-keeper-01 clickhouse-keeper[3460]: Logging trace to /var/log/clickhouse-keeper/clickhouse-keeper.log
Oct 27 19:44:26 clickhouse-keeper-01 clickhouse-keeper[3460]: Logging errors to /var/log/clickhouse-keeper/clickhouse-keeper.err.log

确认clickhouse-keeper-02服务运行状态

root@clickhouse-keeper-02:~# systemctl status clickhouse-keeper.service 
● clickhouse-keeper.service - ClickHouse Keeper - zookeeper compatible distributed coordination server
     Loaded: loaded (/lib/systemd/system/clickhouse-keeper.service; enabled; vendor preset: enabled)
     Active: active (running) since Sun 2024-10-27 19:44:28 CST; 3h 0min ago
   Main PID: 3053 (clickhouse-keep)
      Tasks: 41 (limit: 4556)
     Memory: 44.7M
        CPU: 1min 557ms
     CGroup: /system.slice/clickhouse-keeper.service
             └─3053 /usr/bin/clickhouse-keeper --config=/etc/clickhouse-keeper/keeper_config.xml --pid-file=/run/clickhouse-keeper/clickhouse-keeper.pid

Oct 27 19:44:28 clickhouse-keeper-02 systemd[1]: Started ClickHouse Keeper - zookeeper compatible distributed coordination server.
Oct 27 19:44:28 clickhouse-keeper-02 clickhouse-keeper[3053]: Processing configuration file '/etc/clickhouse-keeper/keeper_config.xml'.
Oct 27 19:44:28 clickhouse-keeper-02 clickhouse-keeper[3053]: Logging trace to /var/log/clickhouse-keeper/clickhouse-keeper.log
Oct 27 19:44:28 clickhouse-keeper-02 clickhouse-keeper[3053]: Logging errors to /var/log/clickhouse-keeper/clickhouse-keeper.err.log

确认clickhouse-keeper-03服务运行状态

root@clickhouse-keeper-03:~# systemctl status clickhouse-keeper.service
● clickhouse-keeper.service - ClickHouse Keeper - zookeeper compatible distributed coordination server
     Loaded: loaded (/lib/systemd/system/clickhouse-keeper.service; enabled; vendor preset: enabled)
     Active: active (running) since Sun 2024-10-27 19:44:30 CST; 3h 0min ago
   Main PID: 2991 (clickhouse-keep)
      Tasks: 41 (limit: 4556)
     Memory: 43.4M
        CPU: 1min 336ms
     CGroup: /system.slice/clickhouse-keeper.service
             └─2991 /usr/bin/clickhouse-keeper --config=/etc/clickhouse-keeper/keeper_config.xml --pid-file=/run/clickhouse-keeper/clickhouse-keeper.pid

Oct 27 19:44:30 clickhouse-keeper-03 systemd[1]: Started ClickHouse Keeper - zookeeper compatible distributed coordination server.
Oct 27 19:44:30 clickhouse-keeper-03 clickhouse-keeper[2991]: Processing configuration file '/etc/clickhouse-keeper/keeper_config.xml'.
Oct 27 19:44:30 clickhouse-keeper-03 clickhouse-keeper[2991]: Logging trace to /var/log/clickhouse-keeper/clickhouse-keeper.log
Oct 27 19:44:30 clickhouse-keeper-03 clickhouse-keeper[2991]: Logging errors to /var/log/clickhouse-keeper/clickhouse-keeper.err.log

在clickhouse-01~02 节点执行；

systemctl enable --now clickhouse-server.service
systemctl restart clickhouse-server.service

确认clickhouse-01服务运行状态

root@clickhouse-01:~# systemctl status clickhouse-server.service 
● clickhouse-server.service - ClickHouse Server (analytic DBMS for big data)
     Loaded: loaded (/lib/systemd/system/clickhouse-server.service; enabled; vendor preset: enabled)
     Active: active (running) since Sun 2024-10-27 19:55:27 CST; 2h 51min ago
   Main PID: 3107 (clickhouse-serv)
      Tasks: 701 (limit: 4556)
     Memory: 802.6M
        CPU: 25min 4.495s
     CGroup: /system.slice/clickhouse-server.service
             ├─3104 clickhouse-watchdog "" "" "" "" "" "" "" --config=/etc/clickhouse-server/config.xml --pid-file=/run/clickhouse-server/clickhouse-server.pid
             └─3107 /usr/bin/clickhouse-server --config=/etc/clickhouse-server/config.xml --pid-file=/run/clickhouse-server/clickhouse-server.pid

Oct 27 19:55:26 clickhouse-01 systemd[1]: Starting ClickHouse Server (analytic DBMS for big data)...
Oct 27 19:55:26 clickhouse-01 clickhouse-server[3104]: Processing configuration file '/etc/clickhouse-server/config.xml'.
Oct 27 19:55:26 clickhouse-01 clickhouse-server[3104]: Merging configuration file '/etc/clickhouse-server/config.d/macros.xml'.
Oct 27 19:55:26 clickhouse-01 clickhouse-server[3104]: Merging configuration file '/etc/clickhouse-server/config.d/network-and-logging.xml'.
Oct 27 19:55:26 clickhouse-01 clickhouse-server[3104]: Merging configuration file '/etc/clickhouse-server/config.d/remote-servers.xml'.
Oct 27 19:55:26 clickhouse-01 clickhouse-server[3104]: Merging configuration file '/etc/clickhouse-server/config.d/use-keeper.xml'.
Oct 27 19:55:26 clickhouse-01 clickhouse-server[3104]: Logging debug to /var/log/clickhouse-server/clickhouse-server.log
Oct 27 19:55:26 clickhouse-01 clickhouse-server[3104]: Logging errors to /var/log/clickhouse-server/clickhouse-server.err.log
Oct 27 19:55:26 clickhouse-01 systemd[1]: clickhouse-server.service: Supervising process 3107 which is not our child. We'll most likely not notice when it exits.
Oct 27 19:55:27 clickhouse-01 systemd[1]: Started ClickHouse Server (analytic DBMS for big data).
root@clickhouse-01:~# 

确认clickhouse-02服务运行状态

root@clickhouse-01:~# systemctl status clickhouse-server.service 
● clickhouse-server.service - ClickHouse Server (analytic DBMS for big data)
     Loaded: loaded (/lib/systemd/system/clickhouse-server.service; enabled; vendor preset: enabled)
     Active: active (running) since Sun 2024-10-27 19:55:27 CST; 2h 51min ago
   Main PID: 3107 (clickhouse-serv)
      Tasks: 701 (limit: 4556)
     Memory: 759.0M
        CPU: 25min 6.801s
     CGroup: /system.slice/clickhouse-server.service
             ├─3104 clickhouse-watchdog "" "" "" "" "" "" "" --config=/etc/clickhouse-server/config.xml --pid-file=/run/clickhouse-server/clickhouse-server.pid
             └─3107 /usr/bin/clickhouse-server --config=/etc/clickhouse-server/config.xml --pid-file=/run/clickhouse-server/clickhouse-server.pid

Oct 27 19:55:26 clickhouse-01 systemd[1]: Starting ClickHouse Server (analytic DBMS for big data)...
Oct 27 19:55:26 clickhouse-01 clickhouse-server[3104]: Processing configuration file '/etc/clickhouse-server/config.xml'.
Oct 27 19:55:26 clickhouse-01 clickhouse-server[3104]: Merging configuration file '/etc/clickhouse-server/config.d/macros.xml'.
Oct 27 19:55:26 clickhouse-01 clickhouse-server[3104]: Merging configuration file '/etc/clickhouse-server/config.d/network-and-logging.xml'.
Oct 27 19:55:26 clickhouse-01 clickhouse-server[3104]: Merging configuration file '/etc/clickhouse-server/config.d/remote-servers.xml'.
Oct 27 19:55:26 clickhouse-01 clickhouse-server[3104]: Merging configuration file '/etc/clickhouse-server/config.d/use-keeper.xml'.
Oct 27 19:55:26 clickhouse-01 clickhouse-server[3104]: Logging debug to /var/log/clickhouse-server/clickhouse-server.log
Oct 27 19:55:26 clickhouse-01 clickhouse-server[3104]: Logging errors to /var/log/clickhouse-server/clickhouse-server.err.log
Oct 27 19:55:26 clickhouse-01 systemd[1]: clickhouse-server.service: Supervising process 3107 which is not our child. We'll most likely not notice when it exits.
Oct 27 19:55:27 clickhouse-01 systemd[1]: Started ClickHouse Server (analytic DBMS for big data).
root@clickhouse-01:~# 

测试集群

要获得 ReplicatedMergeTree 和 ClickHouse Keeper 的经验，您可以运行以下命令：

• 在上面配置的集群上创建数据库
• 使用 ReplicatedMergeTree 表引擎在数据库上创建表
• 在一个节点上插入数据，在另一个节点上查询
• 停止一个ClickHouse服务器节点
• 在运行节点上插入更多数据
• 重新启动停止的节点
• 查询重启节点时验证数据是否可用

验证 ClickHouse Keeper 是否正在运行

mntr命令用于验证 ClickHouse Keeper 是否正在运行并获取有关三个 Keeper 节点关系的状态信息。在此示例中使用的配置中，三个节点一起工作。节点将选举领导者，其余节点将成为追随者。 mntr命令提供与性能以及特定节点是跟随者还是领导者相关的信息。

提示

您可能需要安装netcat才能将mntr命令发送到 Keeper。请参阅nmap.org页面以获取下载信息。

从 clickhouse-keeper-01、clickhouse-keeper-02 和 clickhouse-keeper-03 上的 shell 运行

echo mntr | nc localhost 9181

来自关注者的回应

zk_version  v23.3.1.2823-testing-46e85357ce2da2a99f56ee83a079e892d7ec3726
zk_avg_latency  0
zk_max_latency  0
zk_min_latency  0
zk_packets_received 0
zk_packets_sent 0
zk_num_alive_connections    0
zk_outstanding_requests 0
zk_server_state follower
zk_znode_count  6
zk_watch_count  0
zk_ephemerals_count 0
zk_approximate_data_size    1271
zk_key_arena_size   4096
zk_latest_snapshot_size 0
zk_open_file_descriptor_count   46
zk_max_file_descriptor_count    18446744073709551615

leader的回应

zk_version  v23.3.1.2823-testing-46e85357ce2da2a99f56ee83a079e892d7ec3726
zk_avg_latency  0
zk_max_latency  0
zk_min_latency  0
zk_packets_received 0
zk_packets_sent 0
zk_num_alive_connections    0
zk_outstanding_requests 0
zk_server_state leader
zk_znode_count  6
zk_watch_count  0
zk_ephemerals_count 0
zk_approximate_data_size    1271
zk_key_arena_size   4096
zk_latest_snapshot_size 0
zk_open_file_descriptor_count   48
zk_max_file_descriptor_count    18446744073709551615
zk_followers    2
zk_synced_followers 2

验证 ClickHouse 集群功能

在一个 shell 中使用clickhouse client连接到节点clickhouse-01 ，并在另一个 shell 中使用clickhouse client端连接到节点clickhouse-02 。

1、在上面配置的集群上创建数据库

在节点 clickhouse-01 或 clickhouse-02 上运行

CREATE DATABASE db1 ON CLUSTER cluster_1S_2R

┌─host──────────┬─port─┬─status─┬─error─┬─num_hosts_remaining─┬─num_hosts_active─┐
│ clickhouse-02 │ 9000 │      0 │       │                   1 │                0 │
│ clickhouse-01 │ 9000 │      0 │       │                   0 │                0 │
└───────────────┴──────┴────────┴───────┴─────────────────────┴──────────────────┘

2、使用 ReplicatedMergeTree 表引擎在数据库上创建表

在节点 clickhouse-01 或 clickhouse-02 上运行

CREATE TABLE db1.table1 ON CLUSTER cluster_1S_2R
(
    `id` UInt64,
    `column1` String
)
ENGINE = ReplicatedMergeTree
ORDER BY id

┌─host──────────┬─port─┬─status─┬─error─┬─num_hosts_remaining─┬─num_hosts_active─┐
│ clickhouse-02 │ 9000 │      0 │       │                   1 │                0 │
│ clickhouse-01 │ 9000 │      0 │       │                   0 │                0 │
└───────────────┴──────┴────────┴───────┴─────────────────────┴──────────────────┘

3、在一个节点上插入数据，在另一个节点上查询

在节点 clickhouse-01 上运行

INSERT INTO db1.table1 (id, column1) VALUES (1, 'abc');

4、查询节点clickhouse-02上的表

在节点 clickhouse-02 上运行

SELECT *
FROM db1.table1

┌─id─┬─column1─┐
│  1 │ abc     │
└────┴─────────┘

5、在另一个节点上插入数据，并在节点clickhouse-01上查询

在节点 clickhouse-02 上运行

INSERT INTO db1.table1 (id, column1) VALUES (2, 'def');

在节点 clickhouse-01 上运行

SELECT *
FROM db1.table1

┌─id─┬─column1─┐
│  1 │ abc     │
└────┴─────────┘
┌─id─┬─column1─┐
│  2 │ def     │
└────┴─────────┘

6、停止一个 ClickHouse 服务器节点 通过运行类似于启动该节点的命令的操作系统命令来停止其中一个 ClickHouse 服务器节点。如果您使用systemctl start启动节点，则使用systemctl stop停止它。

root@clickhouse-01:~# systemctl stop clickhouse-server.service

7、在运行节点上插入更多数据

在正在运行的节点上运行

INSERT INTO db1.table1 (id, column1) VALUES (3, 'ghi');

选择数据：

在正在运行的节点上运行

SELECT *
FROM db1.table1

┌─id─┬─column1─┐
│  1 │ abc     │
└────┴─────────┘
┌─id─┬─column1─┐
│  2 │ def     │
└────┴─────────┘
┌─id─┬─column1─┐
│  3 │ ghi     │
└────┴─────────┘

8、重新启动停止的节点并从那里选择

root@clickhouse-01:~# systemctl start clickhouse-server.service

在重启的节点上运行

SELECT *
FROM db1.table1

┌─id─┬─column1─┐
│  1 │ abc     │
└────┴─────────┘
┌─id─┬─column1─┐
│  2 │ def     │
└────┴─────────┘
┌─id─┬─column1─┐
│  3 │ ghi     │
└────┴─────────┘