以下是doris部署的命令全过程,为简略版,没有贴相关图表数据,后面再编辑补充
部署foundationdb=======================
下载foundationdb
安装,会生成foundationdb用户
sudo rpm -Uvh foundationdb-clients-7.1.53-1.el7.x86_64.rpm
sudo rpm -Uvh foundationdb-server-7.1.53-1.el7.x86_64.rpm
配置
修改/etc/foundation/fdb.cluster
JiQOt6wa:Tb2aS9fH@10.7.156.207:4500
修改
可修改datadir = /var/lib/foundationdb/data/$ID,memory等
启动停止
sudo systemctl start foundationdb
sudo systemctl stop foundationdb
验证
fdbcli --exec "status details" 显示activing
fdbcli 可以直接进入shell,之后执行status details
创建doris账户
sudo useradd -m -s /bin/bash doris
修改doris home为doris owner
chown -R doris:doris /data/dis/
添加doris密码
passwd doris
部署ms===================================以下操作在doris账户下操作
下载doris安装包
wget https://apache-doris-releases.oss-accelerate.aliyuncs.com/apache-doris-3.1.4-bin-x64.tar.gz
下载jdk17并解压
将SeagateCA02.cer证书导入jdk17
解压doris包进入ms目录
修改conf/doris_cloud.conf
brpc_listen_port = 5000
fdb_cluster = JiQOt6wa:Tb2aS9fH@10.7.156.207:4500 设置为foundationdb的地址fdb_cluster
修改bin/start.sh
在首行添加java环境为17
export JAVA_HOME=/data/dis/jdk
export PATH=JAVA_HOME/bin:PATH
启动
bin/start.sh --daemon
查看log ms是否启动
部署recycler数据回收功能===================================以下操作在doris账户下操作
在doris home创建recycler目录
copy ms文件夹到recycler目录
修改conf/doris_cloud.conf
修改端口为5001
brpc_listen_port = 5001
fdb_cluster = JiQOt6wa:Tb2aS9fH@10.7.156.207:4500
启动
recycler/ms/bin/start.sh --daemon
查看log ms是否启动
修改文件句柄===================================以下操作在root账户下操作
修改文件数(可用root操作)
/etc/security/limits.conf
最后添加
* soft nofile 655350
* hard nofile 655350
* soft nproc 655350
* hard nproc 655350
关闭当前,重新打开新的session
修改vm map
vi /etc/sysctl.conf
添加
vm.max_map_count = 2000000
加载配置
sysctl -p
禁用swap
临时
sudo swapoff -a
永久
sudo vi /etc/fstab
找到带有 swap 关键字的那一行,注释掉
保存
部署fe===================================以下操作在doris账户下操作
进入fe目录
修改conf/fe.conf,添加
#!/bin/bash
export JAVA_HOME=/data/dis/jdk
export PATH=JAVA_HOME/bin:PATH
JAVA_HOME=/data/dis/jdk
deploy_mode=cloud #支持存算分离
cluster_id=389677793 #随机生成后填入
meta_service_endpoint=10.7.156.207:5000 #ms服务
启动
bin/start_fe.sh --daemon
启动后查询log会有很多报错,需要创建STORAGE VAULT
创建STORAGE VAULT===================================
使用dbeaver的mysql connector,链接fe,ip:10.7.156.207,port:9030
执行sql创建STORAGE VAULT
CREATE STORAGE VAULT IF NOT EXISTS s3_vault
PROPERTIES (
"type"="S3",
"s3.endpoint"="https://korpiibxa03b-s3.seagate.com:9021",
"s3.access_key" = "tnuv1rsj2H1RQFDBM9nr",
"s3.secret_key" = "pSne6XZ2NAHMHoZNFvZSQ8gGYUcjtCPheUdRFUMN",
"s3.region" = "us-east-1",
"s3.root.path" = "ssb_sf1_p2_s3",
"s3.bucket" = "kor-gfit-dev-uss01",
"provider" = "S3"
);
设置STORAGE VAULT为默认
SET s3_vault AS DEFAULT STORAGE VAULT
部署be===================================以下操作在doris账户下操作
在doris home创建filc_cache目录
进入be目录
修改conf/fe.conf,添加
export JAVA_HOME=/data/dis/jdk
export PATH=JAVA_HOME/bin:PATH
JAVA_HOME=/data/dis/jdk
deploy_mode=cloud
file_cache_path=[{"path":"/data/dis/apache-doris-3.1.4-bin-x64/file_cache","total_size":21474836480}]
meta_service_endpoint=10.7.156.207:5000
cloud_unique_id = be_node_01
启动
bin/start_be.sh --daemon
在mysql执行以下语句加入fe
ALTER SYSTEM ADD BACKEND "10.7.156.207:9050"
部署完毕,可以创建数据库,建表,插入数据了
CREATE DATABASE summit;
use summit;
CREATE TABLE IF NOT EXISTS attr_vals (
event_date DATEV2 NOT NULL,
family VARCHAR(50) NOT NULL,
operation VARCHAR(50) NOT NULL,
serial_num VARCHAR(100) NOT NULL,
trans_seq VARCHAR(100),
seq VARCHAR(20),
spc_id VARCHAR(50),
test_seq_event VARCHAR(100),
state_name VARCHAR(100),
occurrence VARCHAR(50),
drive_sbr_num VARCHAR(100),
event_status VARCHAR(50),
birth_date VARCHAR(50),
part_num VARCHAR(50),
drive_event_date VARCHAR(50),
fail_code VARCHAR(50),
attr_name VARCHAR(100),
pre_attr_value VARCHAR(255),
post_attr_value VARCHAR(255),
in_run_file VARCHAR(100)
)
ENGINE=olap
DUPLICATE KEY(event_date, family, operation)
-- 修正点:改为 RANGE 分区,即使字段是 VARCHAR 也支持
PARTITION BY RANGE (event_date) ()
DISTRIBUTED BY HASH(serial_num) BUCKETS AUTO
PROPERTIES (
"replication_num" = "1",
"dynamic_partition.enable" = "true",
"dynamic_partition.time_unit" = "DAY",
"dynamic_partition.start" = "-30", -- 建议改大一点,自动创建过去30天的分区
"dynamic_partition.end" = "3",
"dynamic_partition.prefix" = "p",
"dynamic_partition.create_history_partition" = "true"
);