kafka学习笔记--节点的服役与退役

本文内容来自尚硅谷B站公开教学视频,仅做个人总结、学习、复习使用,任何对此文章的引用,应当说明源出处为尚硅谷,不得用于商业用途。

如有侵权、联系速删

视频教程链接:【尚硅谷】Kafka3.x教程(从入门到调优,深入全面)

文章目录

服役新节点

依旧使用前面文章创建的三个hadoop102、103、104三个节点,模拟节点的服役与退役

新节点准备

(1)关闭 hadoop104,并右键执行克隆操作。

(2)开启 hadoop105,并修改 IP 地址。

powershell 复制代码
[root@hadoop104 ~]# vim /etc/sysconfig/network-scripts/ifcfg-ens33
DEVICE=ens33
TYPE=Ethernet
ONBOOT=yes
BOOTPROTO=static
NAME="ens33"
IPADDR=192.168.10.105
PREFIX=24
GATEWAY=192.168.10.2
DNS1=192.168.10.2

(3)在 hadoop105 上,修改主机名称为 hadoop105。

powershell 复制代码
[root@hadoop104 ~]# vim /etc/hostname
hadoop105

(4)重新启动 hadoop104、hadoop105。

(5)修改 haodoop105 中 kafka 的 broker.id 为 3。

(6)删除 hadoop105 中 kafka 下的 datas 和 logs。

powershell 复制代码
[atguigu@hadoop105 kafka]$ rm -rf datas/* logs/*

(7)启动 hadoop102、hadoop103、hadoop104 上的 kafka 集群。

powershell 复制代码
[atguigu@hadoop102 ~]$ zk.sh start
[atguigu@hadoop102 ~]$ kf.sh start

(8)单独启动 hadoop105 中的 kafka。

powershell 复制代码
[atguigu@hadoop105 kafka]$ bin/kafka-server-start.sh -daemon ./config/server.properties

执行负载均衡操作

(1)创建一个要均衡的主题。

powershell 复制代码
[atguigu@hadoop102 kafka]$ vim topics-to-move.json
{
 "topics": [
 {"topic": "first"}
 ],
 "version": 1
}

(2)生成一个负载均衡的计划。

powershell 复制代码
[atguigu@hadoop102 kafka]$ bin/kafka-reassign-partitions.sh --bootstrap-server hadoop102:9092 --topics-to-move-json-file topics-to-move.json --broker-list "0,1,2,3" --generate
Current partition replica assignment
{"version":1,"partitions":[{"topic":"first","partition":0,"replicas":[0,2,1],"log_dirs":["any","any","any"]},{"topic":"first","partition":1,"replicas":[2,1,0],"log_dirs":["any","any","any"]},{"topic":"first","partition":2,"replicas":[1,0,2],"log_dirs":["any","any","any"]}]}

Proposed partition reassignment configuration
{"version":1,"partitions":[{"topic":"first","partition":0,"replicas":[2,3,0],"log_dirs":["any","any","any"]},{"topic":"first","partition":1,"replicas":[3,0,1],"log_dirs":["any","any","any"]},{"topic":"first","partition":2,"replicas":[0,1,2],"log_dirs":["any","any","any"]}]}

(3)创建副本存储计划(所有副本存储在 broker0、broker1、broker2、broker3 中)。

powershell 复制代码
[atguigu@hadoop102 kafka]$ vim increase-replication-factor.json

输入如下内容:

powershell 复制代码
{"version":1,"partitions":[{"topic":"first","partition":0,"replicas":[2,3,0],"log_dirs":["any","any","any"]},{"topic":"first","partition":1,"replicas":[3,0,1],"log_dirs":["any","any","any"]},{"topic":"first","partition":2,"replicas":[0,1,2],"log_dirs":["any","any","any"]}]}

(4)执行副本存储计划。

powershell 复制代码
[atguigu@hadoop102 kafka]$ bin/kafka-reassign-partitions.sh --bootstrap-server hadoop102:9092 --reassignment-json-file increase-replication-factor.json --execute

(5)验证副本存储计划。

powershell 复制代码
[atguigu@hadoop102 kafka]$ bin/kafka-reassign-partitions.sh --bootstrap-server hadoop102:9092 --reassignment-json-file increase-replication-factor.json --verify

Status of partition reassignment:
Reassignment of partition first-0 is complete.
Reassignment of partition first-1 is complete.
Reassignment of partition first-2 is complete.

Clearing broker-level throttles on brokers 0,1,2,3
Clearing topic-level throttles on topic first

退役旧节点

执行负载均衡操作

先退役一台节点,生成执行计划,然后按照服役时操作流程执行负载均衡。

(1)创建一个要均衡的主题。

powershell 复制代码
[atguigu@hadoop102 kafka]$ vim topics-to-move.json
{
 "topics": [
 {"topic": "first"}
 ],
 "version": 1
}

(2)创建执行计划。

powershell 复制代码
[atguigu@hadoop102 kafka]$ bin/kafka-reassign-partitions.sh --bootstrap-server hadoop102:9092 --topics-to-move-json-file topics-to-move.json --broker-list "0,1,2" --generate

Current partition replica assignment
{"version":1,"partitions":[{"topic":"first","partition":0,"replicas":[2,0,1],"log_dirs":["any","any","any"]},{"topic":"first","partition":1,"replicas":[3,1,2],"log_dirs":["any","any","any"]},{"topic":"first","partition":2,"replicas":[0,2,3],"log_dirs":["any","any","any"]}]}

Proposed partition reassignment configuration
{"version":1,"partitions":[{"topic":"first","partition":0,"replicas":[2,0,1],"log_dirs":["any","any","any"]},{"topic":"first","partition":1,"replicas":[0,1,2],"log_dirs":["any","any","any"]},{"topic":"first","partition":2,"replicas":[1,2,0],"log_dirs":["any","any","any"]}]}

(3)创建副本存储计划(所有副本存储在 broker0、broker1、broker2 中)。

powershell 复制代码
[atguigu@hadoop102 kafka]$ vim increase-replication-factor.json

{"version":1,"partitions":[{"topic":"first","partition":0,"replicas":[2,0,1],"log_dirs":["any","any","any"]},{"topic":"first","partition":1,"replicas":[0,1,2],"log_dirs":["any","any","any"]},{"topic":"first","partition":2,"replicas":[1,2,0],"log_dirs":["any","any","any"]}]}

(4)执行副本存储计划。

powershell 复制代码
[atguigu@hadoop102 kafka]$ bin/kafka-reassign-partitions.sh --bootstrap-server hadoop102:9092 --reassignment-json-file increase-replication-factor.json --execute

(5)验证副本存储计划。

powershell 复制代码
[atguigu@hadoop102 kafka]$ bin/kafka-reassign-partitions.sh --bootstrap-server hadoop102:9092 --reassignment-json-file increase-replication-factor.json --verify

Status of partition reassignment:
Reassignment of partition first-0 is complete.
Reassignment of partition first-1 is complete.
Reassignment of partition first-2 is complete.

Clearing broker-level throttles on brokers 0,1,2,3
Clearing topic-level throttles on topic first

执行停止命令

在 hadoop105 上执行停止命令即可。

powershell 复制代码
[atguigu@hadoop105 kafka]$ bin/kafka-server-stop.sh
相关推荐
受之以蒙7 分钟前
Rust & WASM 之 wasm-bindgen 基础:让 Rust 与 JavaScript 无缝对话
前端·笔记·rust
茫忙然1 小时前
【WEB】Polar靶场 6-10题 详细笔记
笔记
eric*16881 小时前
尚硅谷张天禹老师课程配套笔记
前端·vue.js·笔记·vue·尚硅谷·张天禹·尚硅谷张天禹
Allen_LVyingbo2 小时前
数智读书笔记系列035《未来医疗:医疗4.0引领第四次医疗产业变革》
人工智能·经验分享·笔记·健康医疗
岑梓铭2 小时前
考研408《计算机组成原理》复习笔记,第三章(3)——多模块存储器
笔记·考研·408·计算机组成原理
菜菜why2 小时前
MSPM0G3507学习笔记(一) 重置版:适配逐飞库的ti板环境配置
笔记·学习·电赛·嵌入式软件·mspm0
夜阑卧听风吹雨,铁马冰河入梦来3 小时前
Spring AI 阿里巴巴学习
人工智能·学习·spring
c7693 小时前
【文献笔记】Automatic Chain of Thought Prompting in Large Language Models
人工智能·笔记·语言模型·论文笔记
板栗焖小鸡3 小时前
STM32-PWM驱动无源蜂鸣器
stm32·学习
开开心心就好3 小时前
批量PDF转换工具,一键转换Word Excel
开发语言·前端·学习·pdf·电脑·word·excel