尚硅谷大数据项目《在线教育之实时数仓》笔记002

视频地址:尚硅谷大数据项目《在线教育之实时数仓》_哔哩哔哩_bilibili

目录

[第06章 数据仓库环境准备](#第06章 数据仓库环境准备)

P006

P007

P008

P009

P010

P011

P012

P013

P014


第06章 数据仓库环境准备

P006

P007

P008

http://node001:16010/master-status

atguigu@node001 \~\]$ start-hbase.sh ![](https://file.jishuzhan.net/article/1705840319574052866/59346980b496c3954869a1a786ed4207.webp)

bash 复制代码
[atguigu@node001 ~]$ start-hbase.sh
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/module/hbase/hbase-2.0.5/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/module/hadoop/hadoop-3.1.3/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
running master, logging to /opt/module/hbase/hbase-2.0.5/logs/hbase-atguigu-master-node001.out
node002: running regionserver, logging to /opt/module/hbase/hbase-2.0.5/bin/../logs/hbase-atguigu-regionserver-node002.out
node003: running regionserver, logging to /opt/module/hbase/hbase-2.0.5/bin/../logs/hbase-atguigu-regionserver-node003.out
node001: running regionserver, logging to /opt/module/hbase/hbase-2.0.5/logs/hbase-atguigu-regionserver-node001.out
[atguigu@node001 ~]$ jpsall
================ node001 ================
3041 DataNode
6579 HMaster
2869 NameNode
3447 NodeManager
6778 HRegionServer
6940 Jps
3646 JobHistoryServer
3806 QuorumPeerMain
================ node002 ================
1746 DataNode
3289 HRegionServer
2074 NodeManager
2444 QuorumPeerMain
1949 ResourceManager
3471 Jps
================ node003 ================
2240 QuorumPeerMain
1938 SecondaryNameNode
3138 HRegionServer
1842 DataNode
2070 NodeManager
3338 Jps
[atguigu@node001 ~]$ 

P009

atguigu@node001 conf\]$ cd /opt/module/hbase/apache-phoenix-5.0.0-HBase-2.0-bin \[atguigu@node001 apache-phoenix-5.0.0-HBase-2.0-bin\]$ bin/sqlline.py node001,node002,node003:2181 \[atguigu@node001 \~\]$ start-hbase.sh \[atguigu@node001 \~\]$ /opt/module/hbase/apache-phoenix-5.0.0-HBase-2.0-bin/bin/sqlline.py node001,node002,node003:2181

bash 复制代码
[atguigu@node001 apache-phoenix-5.0.0-HBase-2.0-bin]$ start-hbase.sh 
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/module/hbase/hbase-2.0.5/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/module/hadoop/hadoop-3.1.3/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
running master, logging to /opt/module/hbase/hbase-2.0.5/logs/hbase-atguigu-master-node001.out
node002: running regionserver, logging to /opt/module/hbase/hbase-2.0.5/logs/hbase-atguigu-regionserver-node002.out
node003: running regionserver, logging to /opt/module/hbase/hbase-2.0.5/logs/hbase-atguigu-regionserver-node003.out
node001: running regionserver, logging to /opt/module/hbase/hbase-2.0.5/logs/hbase-atguigu-regionserver-node001.out
node002: running master, logging to /opt/module/hbase/hbase-2.0.5/logs/hbase-atguigu-master-node002.out
[atguigu@node001 apache-phoenix-5.0.0-HBase-2.0-bin]$ bin/sqlline.py node001,node002,node003:2181
Setting property: [incremental, false]
Setting property: [isolation, TRANSACTION_READ_COMMITTED]
issuing: !connect jdbc:phoenix:node001,node002,node003:2181 none none org.apache.phoenix.jdbc.PhoenixDriver
Connecting to jdbc:phoenix:node001,node002,node003:2181
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/module/hbase/apache-phoenix-5.0.0-HBase-2.0-bin/phoenix-5.0.0-HBase-2.0-client.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/module/hadoop/hadoop-3.1.3/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
23/09/12 10:52:47 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Connected to: Phoenix (version 5.0)
Driver: PhoenixEmbeddedDriver (version 5.0)
Autocommit status: true
Transaction isolation: TRANSACTION_READ_COMMITTED
Building list of tables and columns for tab-completion (set fastconnect to true to skip)...
133/133 (100%) Done
Done
sqlline version 1.2.0
0: jdbc:phoenix:node001,node002,node003:2181> !table
+------------+--------------+-------------+---------------+----------+------------+----------------------------+-----------------+--------------+--------------+
| TABLE_CAT  | TABLE_SCHEM  | TABLE_NAME  |  TABLE_TYPE   | REMARKS  | TYPE_NAME  | SELF_REFERENCING_COL_NAME  | REF_GENERATION  | INDEX_STATE  | IMMUTABLE_RO |
+------------+--------------+-------------+---------------+----------+------------+----------------------------+-----------------+--------------+--------------+
|            | SYSTEM       | CATALOG     | SYSTEM TABLE  |          |            |                            |                 |              | false        |
|            | SYSTEM       | FUNCTION    | SYSTEM TABLE  |          |            |                            |                 |              | false        |
|            | SYSTEM       | LOG         | SYSTEM TABLE  |          |            |                            |                 |              | true         |
|            | SYSTEM       | SEQUENCE    | SYSTEM TABLE  |          |            |                            |                 |              | false        |
|            | SYSTEM       | STATS       | SYSTEM TABLE  |          |            |                            |                 |              | false        |
+------------+--------------+-------------+---------------+----------+------------+----------------------------+-----------------+--------------+--------------+
0: jdbc:phoenix:node001,node002,node003:2181> 

P010

6.2.2 Hbase 环境搭建

P011

6.2.3 Redis 环境搭建

安装 redis执行make命令报错struct redisServer'没有名为'sentinel_mode'的成员_server.c:2860:11: 错误:'struct redisserver'没有名为'stat_橙-极纪元的博客-CSDN博客

bash 复制代码
[atguigu@node001 redis-6.0.8]$ /usr/local/bin/redis-server
3651:C 14 Sep 2023 10:23:44.668 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
3651:C 14 Sep 2023 10:23:44.668 # Redis version=6.0.8, bits=64, commit=00000000, modified=0, pid=3651, just started
3651:C 14 Sep 2023 10:23:44.668 # Warning: no config file specified, using the default config. In order to specify a config file use /usr/local/bin/redis-server /path/to/redis.conf
                _._                                                  
           _.-``__ ''-._                                             
      _.-``    `.  `_.  ''-._           Redis 6.0.8 (00000000/0) 64 bit
  .-`` .-```.  ```\/    _.,_ ''-._                                   
 (    '      ,       .-`  | `,    )     Running in standalone mode
 |`-._`-...-` __...-.``-._|'` _.-'|     Port: 6379
 |    `-._   `._    /     _.-'    |     PID: 3651
  `-._    `-._  `-./  _.-'    _.-'                                   
 |`-._`-._    `-.__.-'    _.-'_.-'|                                  
 |    `-._`-._        _.-'_.-'    |           http://redis.io        
  `-._    `-._`-.__.-'_.-'    _.-'                                   
 |`-._`-._    `-.__.-'    _.-'_.-'|                                  
 |    `-._`-._        _.-'_.-'    |                                  
  `-._    `-._`-.__.-'_.-'    _.-'                                   
      `-._    `-.__.-'    _.-'                                       
          `-._        _.-'                                           
              `-.__.-'                                               

3651:M 14 Sep 2023 10:23:44.671 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
3651:M 14 Sep 2023 10:23:44.671 # Server initialized
3651:M 14 Sep 2023 10:23:44.671 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
3651:M 14 Sep 2023 10:23:44.671 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo madvise > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled (set to 'madvise' or 'never').
3651:M 14 Sep 2023 10:23:44.671 * Ready to accept connections

atguigu@node001 redis-6.0.8\]$ /usr/local/bin/redis-server # 前台启动 \[atguigu@node001 \~\]$ redis-server ./my_redis.conf # 后台启动 \[atguigu@node001 \~\]$ jps 4563 Jps \[atguigu@node001 \~\]$ ps -ef \| grep redis atguigu 4558 1 0 10:29 ? 00:00:00 redis-server 127.0.0.1:6379 atguigu 4579 3578 0 10:29 pts/0 00:00:00 grep --color=auto redis \[atguigu@node001 \~\]$ redis-cli 127.0.0.1:6379\> quit \[atguigu@node001 \~\]$

P012

修改redis配置,允许外部访问。

P013

6.2.4 ClickHouse环境搭建
'abrt-cli status' timed out 解决办法(2021综合整理)_abrt-cli status' timed out_爱喝咖啡的程序猿的博客-CSDN博客
cd /-bash: 无法为立即文档创建临时文件: 设备上没有空间。

root@node001 mapper\]# docker stop $(docker ps -aq) # 停止所有的docker容器

bash 复制代码
[atguigu@node001 clickhouse]$ sudo systemctl start clickhouse-server
[atguigu@node001 clickhouse]$ clickhouse-client -m
ClickHouse client version 20.4.5.36 (official build).
Connecting to localhost:9000 as user default.
Code: 209. DB::NetException: Timeout exceeded while reading from socket ([::1]:9000)

[atguigu@node001 clickhouse]$ 
bash 复制代码
[atguigu@node001 ~]$ sudo systemctl status clickhouse-server
● clickhouse-server.service - ClickHouse Server (analytic DBMS for big data)
   Loaded: loaded (/etc/systemd/system/clickhouse-server.service; enabled; vendor preset: disabled)
   Active: active (running) since 二 2023-09-19 10:18:17 CST; 1min 42s ago
 Main PID: 1033 (clickhouse-serv)
    Tasks: 56
   Memory: 230.4M
   CGroup: /system.slice/clickhouse-server.service
           └─1033 /usr/bin/clickhouse-server --config=/etc/clickhouse-server/config.xml --pid-file=/run/clickhouse-server/clickhouse-server.pid

9月 19 10:18:19 node001 clickhouse-server[1033]: Include not found: clickhouse_compression
9月 19 10:18:19 node001 clickhouse-server[1033]: Logging trace to /var/log/clickhouse-server/clickhouse-server.log
9月 19 10:18:19 node001 clickhouse-server[1033]: Logging errors to /var/log/clickhouse-server/clickhouse-server.err.log
9月 19 10:18:20 node001 clickhouse-server[1033]: Processing configuration file '/etc/clickhouse-server/users.xml'.
9月 19 10:18:20 node001 clickhouse-server[1033]: Include not found: networks
9月 19 10:18:20 node001 clickhouse-server[1033]: Saved preprocessed configuration to '/var/lib/clickhouse//preprocessed_configs/users.xml'.
9月 19 10:18:24 node001 clickhouse-server[1033]: Processing configuration file '/etc/clickhouse-server/config.xml'.
9月 19 10:18:24 node001 clickhouse-server[1033]: Include not found: clickhouse_remote_servers
9月 19 10:18:24 node001 clickhouse-server[1033]: Include not found: clickhouse_compression
9月 19 10:18:24 node001 clickhouse-server[1033]: Saved preprocessed configuration to '/var/lib/clickhouse//preprocessed_configs/config.xml'.
[atguigu@node001 ~]$ clickhouse-client -m
ClickHouse client version 20.4.5.36 (official build).
Connecting to localhost:9000 as user default.
Connected to ClickHouse server version 20.4.5 revision 54434.

node001 :) 
node001 :) Bye.
[atguigu@node001 ~]$ 

atguigu@node001 \~\]$ clickhouse-client -m ClickHouse client version 20.4.5.36 (official build). Connecting to localhost:9000 as user default. Connected to ClickHouse server version 20.4.5 revision 54434. node001 :) node001 :) Bye. \[atguigu@node001 \~\]$

P014

6.3 模拟数据准备

bash 复制代码
[atguigu@node001 ~]$ kafka-console-consumer.sh --bootstrap-server node001:9092 --topic topic_log # 启动一个kafka消费者挂起在node001
bash 复制代码
/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/module/data_mocker/01-onlineEducation/edu2021-mock-2022-06-18.jar!/BOOT-INF/lib/slf4j-log4j12-1.7.7.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [ch.qos.logback.classic.util.ContextSelectorStaticBinder]
bash 复制代码
[atguigu@node001 ~]$ kafka-console-consumer.sh --bootstrap-server node001:9092 --topic topic_db
相关推荐
G皮T1 小时前
【Elasticsearch】映射:fielddata 详解
大数据·elasticsearch·搜索引擎·映射·搜索·mappings·fielddata
viperrrrrrrrrr72 小时前
大数据学习(132)-HIve数据分析
大数据·hive·学习
夜影风3 小时前
大数据清洗加工概述
大数据
余+185381628004 小时前
短视频矩阵系统文案创作功能开发实践,定制化开发
大数据·人工智能
阿里云大数据AI技术7 小时前
一体系数据平台的进化:基于阿里云 EMR Serverless Spark的持续演进
大数据·spark·serverless
TDengine (老段)8 小时前
TDengine 开发指南—— UDF函数
java·大数据·数据库·物联网·数据分析·tdengine·涛思数据
Stanford_11068 小时前
关于大数据的基础知识(二)——国内大数据产业链分布结构
大数据·开发语言·物联网·微信小程序·微信公众平台·twitter·微信开放平台
G皮T10 小时前
【Elasticsearch】一个图书馆的案例解释 Elasticsearch
大数据·elasticsearch·搜索引擎·全文检索·kibana·索引·index
cdsmjt11 小时前
当文化遇见科技:探秘国际数字影像创新生态高地
大数据