Clickhouse集群部署

1、环境说明：

测试环境

zk：1x2.x1.22x.231,1x2.x1.22x.232,1x2.x1.22x.233

clickhouse，版本21.8.15.7

shard1 1x2.x1.24x.27 1x2.x1.24x.47

shard2 1x2.x1.24x.98 1x2.x1.24x.210

shard3 1x2.x1.24x.166 1x2.x1.24x.191

ops：172.21.228.116

2、环境准备

登录ops机，编辑ansible配置

vi /etc/ansible/hck

all

172.21.228.116 # ops

ck

1x2.x1.24x.27

1x2.x1.24x.98

1x2.x1.24x.166

ck1

1x2.x1.24x.47

1x2.x1.24x.210

1x2.x1.24x.191

打通ssh免密登录

ansible -i /etc/ansible/hck ck -m authorized_key -a " user=root key="{{ lookup('file', '/root/.ssh/id_rsa.pub') }} " " -u root --become-method=sudo --ask-become-pass --become -k

下载安装包

下载脚步：

dlck.sh

LATEST_VERSION="21.8.15.7";

case $(uname -m) in

x86_64) ARCH=amd64 ;;

aarch64) ARCH=arm64 ;;

*) echo "Unknown architecture (uname−m)";exit1;;esaccd/data/pkgsforPKGinclickhouse−common−staticclickhouse−common−static−dbgclickhouse−serverclickhouse−clientdocurl−fO"https://packages.clickhouse.com/tgz/stable/(uname -m)"; exit 1 ;; esac cd /data/pkgs for PKG in clickhouse-common-static clickhouse-common-static-dbg clickhouse-server clickhouse-client do curl -fO "https://packages.clickhouse.com/tgz/stable/(uname−m)";exit1;;esaccd/data/pkgsforPKGinclickhouse−common−staticclickhouse−common−static−dbgclickhouse−serverclickhouse−clientdocurl−fO"https://packages.clickhouse.com/tgz/stable/PKG-LATESTVERSION−LATEST_VERSION-LATESTVERSION−{ARCH}.tgz"

|| curl -fO "https://packages.clickhouse.com/tgz/stable/PKG−PKG-PKG−LATEST_VERSION.tgz"

done

sh dlck.sh

拷贝安装包到所有节点

vi cp.yaml

cp.yaml

hosts: ck,ck1
vars:
filename: /data/pkgs/clickhouse*
tasks:
- copy:
  src: "{{ item }}"
  dest: /data/
  with_file:
  - "{{ filename }}"

ansible-playbook cp.yaml -i /etc/ansible/hck

3、安装clickhouse

登录第1点的节点表的所有节点（如1x2.x1.24x.27），逐个重复下面安装步骤：

工具解压和配置

tar -xzvf clickhouse-common-static-21.8.15.7.tgz

tar -xzvf clickhouse-common-static-dbg-21.8.15.7.tgz

tar -xzvf clickhouse-server-21.8.15.7.tgz

clickhouse-common-static-21.8.15.7/install/doinst.sh

clickhouse-common-static-dbg-21.8.15.7/install/doinst.sh

sudo "clickhouse-server-21.8.15.7/install/doinst.sh" configure

注意最后一步有交互输入，第一次输入defualt帐号的密码，第二次输入yes

修改配置文件并启动服务

cd /etc/clickhouse-server/

编辑用户配置文件

vi user.xml

user.xml 展开源码

编辑全局配置文件

vi config.xml

config.xml 展开源码

编辑集群配置文件

vi config.d/cluster.xml

cluster.xml 展开源码

按照实际的集群拓扑和zookeeper集群信息配置，注意 1x2.x1.24x.27 这个片段每个节点不一样，都当前节点的ip

chown clickhouse:clickhouse -R /data

chown clickhouse:clickhouse -R /etc/clickhouse-server/

登陆ops机启动集群

ansible -i /etc/ansible/hck ck,ck1 -m shell -a " systemctl start clickhouse-server "

4、建立分布式库表和帐号

建库建表建帐号

CREATE DATABASE IF NOT EXISTS birept ON CLUSTER ckc_test;

CREATE TABLE IF NOT EXISTS birept.bi_event_local

ON CLUSTER ckc_test(
proj String COMMENT '项目CODE',
ptype Nullable(String) COMMENT '二级分类',
ip Nullable(String) COMMENT 'IP地址',
distinct_id String COMMENT '用户唯一标识',
anonymous_id String COMMENT '用户匿名标识',
login_id Nullable(String) COMMENT '业务系统登录后的用户ID',
d_time UInt64 COMMENT '时间',
d_timestamp DateTime COMMENT '时间',
d_date Date COMMENT '时间日期型',
d_type Nullable(String) COMMENT '事件类型',
event String COMMENT '事件',
lib Nullable(String) COMMENT '引用',
screen_width Nullable(Int16) COMMENT '屏幕宽度',
screen_height Nullable(Int16) COMMENT '屏幕高度',
_track_id Nullable(Int32) COMMENT '跟踪ID',
platform_type Nullable(String) COMMENT '平台类型',
tenant_id Nullable(String) COMMENT '租户ID',
attr1 Nullable(String) COMMENT '属性1',
attr2 Nullable(String) COMMENT '属性2',
attr3 Nullable(String) COMMENT '属性3',
attr4 Nullable(String) COMMENT '属性4',
attr5 Nullable(String) COMMENT '属性5',
attr6 Nullable(String) COMMENT '属性6',
attr7 Nullable(String) COMMENT '属性7',
attr8 Nullable(String) COMMENT '属性8',
attr9 Nullable(String) COMMENT '属性9',
attr10 Nullable(String) COMMENT '属性10',
attr11 Nullable(String) COMMENT '属性11',
attr12 Nullable(String) COMMENT '属性12',
attr13 Nullable(String) COMMENT '属性13',
attr14 Nullable(String) COMMENT '属性14',
attr15 Nullable(String) COMMENT '属性15',
attr16 Nullable(String) COMMENT '属性16',
attr17 Nullable(String) COMMENT '属性17',
attr18 Nullable(String) COMMENT '属性18',
attr19 Nullable(String) COMMENT '属性19',
attr20 Nullable(String) COMMENT '属性20'

)

ENGINE = ReplicatedMergeTree('/clickhouse/tables/{shard}/default/bi_event_local', '{replica}')

ORDER BY (proj, d_time, d_timestamp, event)

SETTINGS index_granularity = 8192;

CREATE TABLE IF NOT EXISTS birept.bi_event

ON CLUSTER ckc_test AS birept.bi_event_local

ENGINE = Distributed(ckc_test, birept, bi_event_local, rand());

CREATE USER IF NOT EXISTS biuser ON CLUSTER ckc_test IDENTIFIED BY 'y3H876dslly';

GRANT ON CLUSTER ckc_test all ON birept.* TO biuser;

附录：systemd的service文件

/etc/systemd/system/clickhouse-server.service

Unit

Description=ClickHouse Server (analytic DBMS for big data)

Requires=network-online.target

NOTE: that After/Wants=time-sync.target is not enough, you need to ensure

that the time was adjusted already, if you use systemd-timesyncd you are

safe, but if you use ntp or some other daemon, you should configure it

additionaly.

After=time-sync.target network-online.target

Wants=time-sync.target

Service

Type=notify

NOTE: we leave clickhouse watchdog process enabled to be able to see OOM/SIGKILL traces in clickhouse-server.log files.

If you wish to disable the watchdog and rely on systemd logs just add "Environment=CLICKHOUSE_WATCHDOG_ENABLE=0" line.

User=clickhouse

Group=clickhouse

Restart=always

RestartSec=30

The following ClickHouse directives should be used instead of forcing SIGKILL by systemd:

- shutdown_wait_unfinished_queries

- shutdown_wait_unfinished

TimeoutStopSec=infinity

Disable forwarding signals by watchdog, since with default systemd's

kill-mode control-group, systemd will send signal to all process in cgroup.

Environment=CLICKHOUSE_WATCHDOG_NO_FORWARD=1

Since ClickHouse is systemd aware default 1m30sec may not be enough

TimeoutStartSec=0

%p is resolved to the systemd unit name

RuntimeDirectory=%p

ExecStart=/usr/bin/clickhouse-server --config=/etc/clickhouse-server/config.xml --pid-file=%t/%p/%p.pid

Minus means that this file is optional.

EnvironmentFile=-/etc/default/%p

Bring back /etc/default/clickhouse for backward compatibility

EnvironmentFile=-/etc/default/clickhouse

LimitCORE=infinity

LimitNOFILE=500000

CapabilityBoundingSet=CAP_NET_ADMIN CAP_IPC_LOCK CAP_SYS_NICE CAP_NET_BIND_SERVICE

AmbientCapabilities=CAP_NET_ADMIN CAP_IPC_LOCK CAP_SYS_NICE CAP_NET_BIND_SERVICE

Install

ClickHouse should not start from the rescue shell (rescue.target).

WantedBy=multi-user.target