尚硅谷大数据项目《在线教育之实时数仓》笔记002

视频地址:尚硅谷大数据项目《在线教育之实时数仓》_哔哩哔哩_bilibili

目录

[第06章 数据仓库环境准备](#第06章 数据仓库环境准备)

P006

P007

P008

P009

P010

P011

P012

P013

P014


第06章 数据仓库环境准备

P006

P007

P008

http://node001:16010/master-status

atguigu@node001 \~\]$ start-hbase.sh ![](https://file.jishuzhan.net/article/1705840319574052866/59346980b496c3954869a1a786ed4207.webp)

bash 复制代码
[atguigu@node001 ~]$ start-hbase.sh
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/module/hbase/hbase-2.0.5/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/module/hadoop/hadoop-3.1.3/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
running master, logging to /opt/module/hbase/hbase-2.0.5/logs/hbase-atguigu-master-node001.out
node002: running regionserver, logging to /opt/module/hbase/hbase-2.0.5/bin/../logs/hbase-atguigu-regionserver-node002.out
node003: running regionserver, logging to /opt/module/hbase/hbase-2.0.5/bin/../logs/hbase-atguigu-regionserver-node003.out
node001: running regionserver, logging to /opt/module/hbase/hbase-2.0.5/logs/hbase-atguigu-regionserver-node001.out
[atguigu@node001 ~]$ jpsall
================ node001 ================
3041 DataNode
6579 HMaster
2869 NameNode
3447 NodeManager
6778 HRegionServer
6940 Jps
3646 JobHistoryServer
3806 QuorumPeerMain
================ node002 ================
1746 DataNode
3289 HRegionServer
2074 NodeManager
2444 QuorumPeerMain
1949 ResourceManager
3471 Jps
================ node003 ================
2240 QuorumPeerMain
1938 SecondaryNameNode
3138 HRegionServer
1842 DataNode
2070 NodeManager
3338 Jps
[atguigu@node001 ~]$ 

P009

atguigu@node001 conf\]$ cd /opt/module/hbase/apache-phoenix-5.0.0-HBase-2.0-bin \[atguigu@node001 apache-phoenix-5.0.0-HBase-2.0-bin\]$ bin/sqlline.py node001,node002,node003:2181 \[atguigu@node001 \~\]$ start-hbase.sh \[atguigu@node001 \~\]$ /opt/module/hbase/apache-phoenix-5.0.0-HBase-2.0-bin/bin/sqlline.py node001,node002,node003:2181

bash 复制代码
[atguigu@node001 apache-phoenix-5.0.0-HBase-2.0-bin]$ start-hbase.sh 
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/module/hbase/hbase-2.0.5/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/module/hadoop/hadoop-3.1.3/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
running master, logging to /opt/module/hbase/hbase-2.0.5/logs/hbase-atguigu-master-node001.out
node002: running regionserver, logging to /opt/module/hbase/hbase-2.0.5/logs/hbase-atguigu-regionserver-node002.out
node003: running regionserver, logging to /opt/module/hbase/hbase-2.0.5/logs/hbase-atguigu-regionserver-node003.out
node001: running regionserver, logging to /opt/module/hbase/hbase-2.0.5/logs/hbase-atguigu-regionserver-node001.out
node002: running master, logging to /opt/module/hbase/hbase-2.0.5/logs/hbase-atguigu-master-node002.out
[atguigu@node001 apache-phoenix-5.0.0-HBase-2.0-bin]$ bin/sqlline.py node001,node002,node003:2181
Setting property: [incremental, false]
Setting property: [isolation, TRANSACTION_READ_COMMITTED]
issuing: !connect jdbc:phoenix:node001,node002,node003:2181 none none org.apache.phoenix.jdbc.PhoenixDriver
Connecting to jdbc:phoenix:node001,node002,node003:2181
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/module/hbase/apache-phoenix-5.0.0-HBase-2.0-bin/phoenix-5.0.0-HBase-2.0-client.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/module/hadoop/hadoop-3.1.3/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
23/09/12 10:52:47 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Connected to: Phoenix (version 5.0)
Driver: PhoenixEmbeddedDriver (version 5.0)
Autocommit status: true
Transaction isolation: TRANSACTION_READ_COMMITTED
Building list of tables and columns for tab-completion (set fastconnect to true to skip)...
133/133 (100%) Done
Done
sqlline version 1.2.0
0: jdbc:phoenix:node001,node002,node003:2181> !table
+------------+--------------+-------------+---------------+----------+------------+----------------------------+-----------------+--------------+--------------+
| TABLE_CAT  | TABLE_SCHEM  | TABLE_NAME  |  TABLE_TYPE   | REMARKS  | TYPE_NAME  | SELF_REFERENCING_COL_NAME  | REF_GENERATION  | INDEX_STATE  | IMMUTABLE_RO |
+------------+--------------+-------------+---------------+----------+------------+----------------------------+-----------------+--------------+--------------+
|            | SYSTEM       | CATALOG     | SYSTEM TABLE  |          |            |                            |                 |              | false        |
|            | SYSTEM       | FUNCTION    | SYSTEM TABLE  |          |            |                            |                 |              | false        |
|            | SYSTEM       | LOG         | SYSTEM TABLE  |          |            |                            |                 |              | true         |
|            | SYSTEM       | SEQUENCE    | SYSTEM TABLE  |          |            |                            |                 |              | false        |
|            | SYSTEM       | STATS       | SYSTEM TABLE  |          |            |                            |                 |              | false        |
+------------+--------------+-------------+---------------+----------+------------+----------------------------+-----------------+--------------+--------------+
0: jdbc:phoenix:node001,node002,node003:2181> 

P010

6.2.2 Hbase 环境搭建

P011

6.2.3 Redis 环境搭建

安装 redis执行make命令报错struct redisServer'没有名为'sentinel_mode'的成员_server.c:2860:11: 错误:'struct redisserver'没有名为'stat_橙-极纪元的博客-CSDN博客

bash 复制代码
[atguigu@node001 redis-6.0.8]$ /usr/local/bin/redis-server
3651:C 14 Sep 2023 10:23:44.668 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
3651:C 14 Sep 2023 10:23:44.668 # Redis version=6.0.8, bits=64, commit=00000000, modified=0, pid=3651, just started
3651:C 14 Sep 2023 10:23:44.668 # Warning: no config file specified, using the default config. In order to specify a config file use /usr/local/bin/redis-server /path/to/redis.conf
                _._                                                  
           _.-``__ ''-._                                             
      _.-``    `.  `_.  ''-._           Redis 6.0.8 (00000000/0) 64 bit
  .-`` .-```.  ```\/    _.,_ ''-._                                   
 (    '      ,       .-`  | `,    )     Running in standalone mode
 |`-._`-...-` __...-.``-._|'` _.-'|     Port: 6379
 |    `-._   `._    /     _.-'    |     PID: 3651
  `-._    `-._  `-./  _.-'    _.-'                                   
 |`-._`-._    `-.__.-'    _.-'_.-'|                                  
 |    `-._`-._        _.-'_.-'    |           http://redis.io        
  `-._    `-._`-.__.-'_.-'    _.-'                                   
 |`-._`-._    `-.__.-'    _.-'_.-'|                                  
 |    `-._`-._        _.-'_.-'    |                                  
  `-._    `-._`-.__.-'_.-'    _.-'                                   
      `-._    `-.__.-'    _.-'                                       
          `-._        _.-'                                           
              `-.__.-'                                               

3651:M 14 Sep 2023 10:23:44.671 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
3651:M 14 Sep 2023 10:23:44.671 # Server initialized
3651:M 14 Sep 2023 10:23:44.671 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
3651:M 14 Sep 2023 10:23:44.671 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo madvise > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled (set to 'madvise' or 'never').
3651:M 14 Sep 2023 10:23:44.671 * Ready to accept connections

atguigu@node001 redis-6.0.8\]$ /usr/local/bin/redis-server # 前台启动 \[atguigu@node001 \~\]$ redis-server ./my_redis.conf # 后台启动 \[atguigu@node001 \~\]$ jps 4563 Jps \[atguigu@node001 \~\]$ ps -ef \| grep redis atguigu 4558 1 0 10:29 ? 00:00:00 redis-server 127.0.0.1:6379 atguigu 4579 3578 0 10:29 pts/0 00:00:00 grep --color=auto redis \[atguigu@node001 \~\]$ redis-cli 127.0.0.1:6379\> quit \[atguigu@node001 \~\]$

P012

修改redis配置,允许外部访问。

P013

6.2.4 ClickHouse环境搭建
'abrt-cli status' timed out 解决办法(2021综合整理)_abrt-cli status' timed out_爱喝咖啡的程序猿的博客-CSDN博客
cd /-bash: 无法为立即文档创建临时文件: 设备上没有空间。

root@node001 mapper\]# docker stop $(docker ps -aq) # 停止所有的docker容器

bash 复制代码
[atguigu@node001 clickhouse]$ sudo systemctl start clickhouse-server
[atguigu@node001 clickhouse]$ clickhouse-client -m
ClickHouse client version 20.4.5.36 (official build).
Connecting to localhost:9000 as user default.
Code: 209. DB::NetException: Timeout exceeded while reading from socket ([::1]:9000)

[atguigu@node001 clickhouse]$ 
bash 复制代码
[atguigu@node001 ~]$ sudo systemctl status clickhouse-server
● clickhouse-server.service - ClickHouse Server (analytic DBMS for big data)
   Loaded: loaded (/etc/systemd/system/clickhouse-server.service; enabled; vendor preset: disabled)
   Active: active (running) since 二 2023-09-19 10:18:17 CST; 1min 42s ago
 Main PID: 1033 (clickhouse-serv)
    Tasks: 56
   Memory: 230.4M
   CGroup: /system.slice/clickhouse-server.service
           └─1033 /usr/bin/clickhouse-server --config=/etc/clickhouse-server/config.xml --pid-file=/run/clickhouse-server/clickhouse-server.pid

9月 19 10:18:19 node001 clickhouse-server[1033]: Include not found: clickhouse_compression
9月 19 10:18:19 node001 clickhouse-server[1033]: Logging trace to /var/log/clickhouse-server/clickhouse-server.log
9月 19 10:18:19 node001 clickhouse-server[1033]: Logging errors to /var/log/clickhouse-server/clickhouse-server.err.log
9月 19 10:18:20 node001 clickhouse-server[1033]: Processing configuration file '/etc/clickhouse-server/users.xml'.
9月 19 10:18:20 node001 clickhouse-server[1033]: Include not found: networks
9月 19 10:18:20 node001 clickhouse-server[1033]: Saved preprocessed configuration to '/var/lib/clickhouse//preprocessed_configs/users.xml'.
9月 19 10:18:24 node001 clickhouse-server[1033]: Processing configuration file '/etc/clickhouse-server/config.xml'.
9月 19 10:18:24 node001 clickhouse-server[1033]: Include not found: clickhouse_remote_servers
9月 19 10:18:24 node001 clickhouse-server[1033]: Include not found: clickhouse_compression
9月 19 10:18:24 node001 clickhouse-server[1033]: Saved preprocessed configuration to '/var/lib/clickhouse//preprocessed_configs/config.xml'.
[atguigu@node001 ~]$ clickhouse-client -m
ClickHouse client version 20.4.5.36 (official build).
Connecting to localhost:9000 as user default.
Connected to ClickHouse server version 20.4.5 revision 54434.

node001 :) 
node001 :) Bye.
[atguigu@node001 ~]$ 

atguigu@node001 \~\]$ clickhouse-client -m ClickHouse client version 20.4.5.36 (official build). Connecting to localhost:9000 as user default. Connected to ClickHouse server version 20.4.5 revision 54434. node001 :) node001 :) Bye. \[atguigu@node001 \~\]$

P014

6.3 模拟数据准备

bash 复制代码
[atguigu@node001 ~]$ kafka-console-consumer.sh --bootstrap-server node001:9092 --topic topic_log # 启动一个kafka消费者挂起在node001
bash 复制代码
/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/module/data_mocker/01-onlineEducation/edu2021-mock-2022-06-18.jar!/BOOT-INF/lib/slf4j-log4j12-1.7.7.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [ch.qos.logback.classic.util.ContextSelectorStaticBinder]
bash 复制代码
[atguigu@node001 ~]$ kafka-console-consumer.sh --bootstrap-server node001:9092 --topic topic_db
相关推荐
qq_463944863 小时前
【Spark征服之路-2.2-安装部署Spark(二)】
大数据·分布式·spark
weixin_505154464 小时前
数字孪生在建设智慧城市中可以起到哪些作用或帮助?
大数据·人工智能·智慧城市·数字孪生·数据可视化
打码人的日常分享4 小时前
智慧城市建设方案
大数据·架构·智慧城市·制造
weixin_307779134 小时前
Linux下GCC和C++实现统计Clickhouse数据仓库指定表中各字段的空值、空字符串或零值比例
linux·运维·c++·数据仓库·clickhouse
阿里云大数据AI技术6 小时前
ES Serverless 8.17王牌发布:向量检索「火力全开」,智能扩缩「秒级响应」!
大数据·运维·serverless
Mikhail_G7 小时前
Python应用变量与数据类型
大数据·运维·开发语言·python·数据分析
G皮T7 小时前
【Elasticsearch】映射:null_value 详解
大数据·elasticsearch·搜索引擎·映射·mappings·null_value
大霸王龙8 小时前
软件工程的软件生命周期通常分为以下主要阶段
大数据·人工智能·旅游
点赋科技9 小时前
沙市区举办资本市场赋能培训会 点赋科技分享智能消费新实践
大数据·人工智能
YSGZJJ9 小时前
股指期货技术分析与短线操作方法介绍
大数据·人工智能