apache druid整合hadoop3.3

安装流程

修改配置文件

  1. 解压文件

    tar -xvf apache-druid-31.0.0-bin.tar.gz

    mv apache-druid-31.0.0 druid

  2. 拷贝hadoop配置文件到druid

    ln -s /app/hadoop-3.3.6/etc/hadoop/core-site.xml /app/druid/conf/druid/cluster/_common/core-site.xml

    ln -s /app/hadoop-3.3.6/etc/hadoop/hdfs-site.xml /app/druid/conf/druid/cluster/_common/hdfs-site.xml

    ln -s /app/hadoop-3.3.6/etc/hadoop/yarn-site.xml /app/druid/conf/druid/cluster/_common/yarn-site.xml

    ln -s /app/hadoop-3.3.6/etc/hadoop/mapred-site.xml /app/druid/conf/druid/cluster/_common/mapred-site.xml

  3. 拷贝连接MYSQL的jar包

    将mysql的jdbc驱动拷贝到/app/druid/extensions/mysql-metadata-storage

  4. 添加druid的mysql用户

    create database druid;

    create user 'druid'@'%' identified by 'druid';

    ALTER USER 'druid'@'%' IDENTIFIED BY 'druid';

    grant all privileges on druid.* to 'druid'@'%' with grant option;

    grant all privileges on mysql.* to 'druid'@'%' with grant option;

  5. 编辑druid的配置文件

    vim /app/druid/conf/druid/cluster/_common/common.runtime.properties

    druid.extensions.loadList=["mysql-metadata-storage","druid-hdfs-storage", "druid-kafka-indexing-service", "druid-datasketches", "druid-multi-stage-query", "druid-parquet-extensions"]

    druid.host=hadoop-drill-nn-1

    druid.zk.service.host=10.10.9.22:2181

    druid.zk.paths.base=/druid

    druid.metadata.storage.type=mysql

    druid.metadata.storage.connector.connectURI=jdbc:mysql://10.10.100.95:3306/druid

    druid.metadata.storage.connector.user=druid

    druid.metadata.storage.connector.password=druid

    druid.storage.type=hdfs

    druid.storage.storageDirectory=/druid/segments

    druid.indexer.logs.type=hdfs

    druid.indexer.logs.directory=/druid/indexing-logs

  6. 修改coordinator-overlord配置(主节点)

    vim /app/druid/conf/druid/cluster/master/coordinator-overlord/jvm.config

    -server

    -Xms2g

    -Xmx2g

    -XX:+ExitOnOutOfMemoryError

    -XX:+UseG1GC

    -Duser.timezone=UTC

    -Dfile.encoding=UTF-8

    -Djava.io.tmpdir=var/tmp

    -Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager

    -Dderby.stream.error.file=var/druid/derby.log

    vim /app/druid/conf/druid/cluster/master/coordinator-overlord/runtime.properties

    druid.coordinator.http.port=18082

  7. 修改historical配置(数据节点)

    vim /app/druid/conf/druid/cluster/data/historical/jvm.config

    -server

    -Xms2g

    -Xmx2g

    -XX:MaxDirectMemorySize=13g

    -XX:+ExitOnOutOfMemoryError

    -Duser.timezone=UTC

    -Dfile.encoding=UTF-8

    -Djava.io.tmpdir=var/tmp

    -Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager

    vim /app/druid/conf/druid/cluster/data/historical/runtime.properties

    #内存不够减小此值

    druid.processing.buffer.sizeBytes=50MiB

    druid.historical.http.port=18083

  8. 修改middleManager(数据节点)

    vim /app/druid/conf/druid/cluster/data/middleManager/jvm.config

    -server

    -Xms128m

    -Xmx128m

    -XX:+ExitOnOutOfMemoryError

    -Duser.timezone=UTC

    -Dfile.encoding=UTF-8

    -Djava.io.tmpdir=var/tmp

    -Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager

    vim /app/druid/conf/druid/cluster/data/middleManager/runtime.properties

    druid.indexer.runner.http.port=18091

  9. 修改broker(查询节点)

    vim /app/druid/conf/druid/cluster/query/broker/jvm.config

    -server

    -Xms1g

    -Xmx1g

    -XX:MaxDirectMemorySize=6g

    -XX:+ExitOnOutOfMemoryError

    -Duser.timezone=UTC

    -Dfile.encoding=UTF-8

    -Djava.io.tmpdir=var/tmp

    -Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager

    vim /app/druid/conf/druid/cluster/query/broker/runtime.properties

    druid.processing.buffer.sizeBytes=50MiB

    druid.broker.http.port=18082

  10. 修改router(查询节点)

    vim /app/druid/conf/druid/cluster/query/router/jvm.config

    vim /app/druid/conf/druid/cluster/query/router/runtime.properties

    druid.router.http.port=18888

启动相关进程

  1. 启动查询节点

    nohup start-cluster-query-server &

  2. 启动数据节点

    nohup start-cluster-data-server &

  3. 启动主节点

    nohup start-cluster-master-no-zk-server &

停止服务

复制代码
/app/druid/bin/service --down

查看运行情况

http://10.10.100.95:8888

相关推荐
智_永无止境2 小时前
Apache Commons Math3 使用指南:强大的Java数学库
apache·math
一只 Lemon12 小时前
jquery 文件上传 (CVE-2018-9207)漏洞复现
apache
喜欢你,还有大家1 天前
Apache服务——搭建实验
apache
一休哥助手2 天前
Apache Thrift:跨语言服务开发的高性能RPC框架指南
网络协议·rpc·apache
Apache IoTDB2 天前
Apache IoTDB V1.3.5 发布|优化加密算法,优化内核稳定性,修复社区反馈问题
apache·iotdb
悠悠~飘2 天前
php学习(第四天)
php·apache
白鹭2 天前
apache实现LAMP+apache(URL重定向)
linux·运维·apache·url重定向·apache实现lamp架构
aramae2 天前
终端之外:解锁Linux命令行的魔法与力量
linux·服务器·apache
lifallen2 天前
Hadoop MapOutputBuffer:Map高性能核心揭秘
java·大数据·数据结构·hadoop·算法·apache
acaad2 天前
Apache Poi 实现导出excel表格 合并区域边框未完全显示的问题
spring·apache·excel