Zab协议 的全称是 Zookeeper Atomic Broadcast (Zookeeper原子广播)
Zookeeper的Zab协议代码十分复杂,涉及多个实例和多线程的调用,通过Aop日志分析才弄明白。
ZAB协议流程
请求流程
核心步骤:
步骤一: 将请求转发给leader,如果客户端所连的服务端是follwer节点
步骤二: leader发送Proposal,给所有follwer节点
步骤三: 处理leader的Proposal,将提交放到pendingTxns,并且发送Ack请求
步骤四: 判断是否成功,发送commit请求,收到大于一半节点的ack则可以提交
步骤五: 所有节点处理提交请求,修改数据库
具体处理逻辑在下面节点分析有更清楚的描述
查看节点状态
powershell命令
bash
echo -----node1------; echo stat | nc 127.0.0.1 2181 ; echo -------node2-----; echo stat | nc 127.0.0.1 12181 ; echo -------node3--------; echo stat | nc 127.0.0.1 22181 ;
状态结果
yaml
-----node1------
Zookeeper version: 3.9.1-SNAPSHOT-73ee39c034ea6eba2d46b0308c0fed68293e30c1-dirty, built on 2023-09-05 11:42 UTC
Clients:
/127.0.0.1:12594[0](queued=0,recved=1,sent=0)
Latency min/avg/max: 0/0.0/0
Received: 1
Sent: 0
Connections: 1
Outstanding: 0
Zxid: 0x0
Mode: follower
Node count: 5
-------node2-----
Zookeeper version: 3.9.1-SNAPSHOT-73ee39c034ea6eba2d46b0308c0fed68293e30c1-dirty, built on 2023-09-05 11:42 UTC
Clients:
/127.0.0.1:12596[0](queued=0,recved=1,sent=0)
Latency min/avg/max: 0/0.0/0
Received: 1
Sent: 014:
Connections: 1
Outstanding: 0
Zxid: 0x200000000
Mode: follower
Node count: 5
-------node3--------
Zookeeper version: 3.9.1-SNAPSHOT-73ee39c034ea6eba2d46b0308c0fed68293e30c1-dirty, built on 2023-09-05 11:42 UTC
Clients:
/127.0.0.1:12598[0](queued=0,recved=1,sent=0)
Latency min/avg/max: 0/0.0/0
Received: 1
Sent: 0
Connections: 1
Outstanding: 0
Zxid: 0x200000000
Mode: leader
Node count: 5
Proposal sizes last/min/max: -1/-1/-1
分析
- 客户端连的是node1节点
- node3是leader节点
执行分析
客户端执行,通过日志分析服务端的执行逻辑
bash
create /data11 1
日志通过aspectJ打印
scss
public aspect ZookeeperAspect
{
pointcut myPointCut(): execution ( public * * (org.apache.zookeeper.server.Request));
before() : myPointCut()
{
System.out.println("时间:"+ java.time.LocalDateTime.now()+" 进入 " + thisJoinPoint );
if(thisJoinPoint.getArgs()!=null&&thisJoinPoint.getArgs().length>0){
System.out.print(" 参数 ");
for(Object obj: thisJoinPoint.getArgs())
{
System.out.print(obj);
}
System.out.println(" 参数结尾 ");
}
}
}
node1
日志
css
时间:2023-09-05T19:45:23.959 进入 execution(void org.apache.zookeeper.server.ZooKeeperServer.submitRequest(Request))
参数 sessionid:0x100064ba8b80000 type:create cxid:0x1 zxid:0xfffffffffffffffe txntype:unknown reqpath:/data11 参数结尾 时间:2023-09-05T19:45:23.959 进入 execution(void org.apache.zookeeper.server.ZooKeeperServer.enqueueRequest(Request))
参数 sessionid:0x100064ba8b80000 type:create cxid:0x1 zxid:0xfffffffffffffffe txntype:unknown reqpath:/data11 参数结尾 时间:2023-09-05T19:45:23.959 进入 execution(void org.apache.zookeeper.server.RequestThrottler.submitRequest(Request))
参数 sessionid:0x100064ba8b80000 type:create cxid:0x1 zxid:0xfffffffffffffffe txntype:unknown reqpath:/data11 参数结尾 时间:2023-09-05T19:45:23.959 进入 execution(void org.apache.zookeeper.server.ZooKeeperServer.submitRequestNow(Request))
参数 sessionid:0x100064ba8b80000 type:create cxid:0x1 zxid:0xfffffffffffffffe txntype:unknown reqpath:/data11 参数结尾 时间:2023-09-05T19:45:23.959 进入 execution(void org.apache.zookeeper.server.quorum.FollowerRequestProcessor.processRequest(Request))
参数 sessionid:0x100064ba8b80000 type:create cxid:0x1 zxid:0xfffffffffffffffe txntype:unknown reqpath:/data11 参数结尾 时间:2023-09-05T19:45:23.960 进入 execution(Request org.apache.zookeeper.server.quorum.QuorumZooKeeperServer.checkUpgradeSession(Request))
参数 sessionid:0x100064ba8b80000 type:create cxid:0x1 zxid:0xfffffffffffffffe txntype:unknown reqpath:/data11 参数结尾 时间:2023-09-05T19:45:23.960 进入 execution(boolean org.apache.zookeeper.server.ZooKeeperServer.authWriteRequest(Request))
参数 sessionid:0x100064ba8b80000 type:create cxid:0x1 zxid:0xfffffffffffffffe txntype:unknown reqpath:/data11 参数结尾 时间:2023-09-05T19:45:23.960 进入 execution(void org.apache.zookeeper.server.quorum.CommitProcessor.processRequest(Request))
参数 sessionid:0x100064ba8b80000 type:create cxid:0x1 zxid:0xfffffffffffffffe txntype:unknown reqpath:/data11 参数结尾 14:45:23.960 [FollowerRequestProcessor:1] DEBUG org.apache.zookeeper.server.quorum.CommitProcessor - Processing request:: sessionid:0x100064ba8b80000 type:create cxid:0x1 zxid:0xfffffffffffffffe txntype:unknown reqpath:/data11
时间:2023-09-05T19:45:23.973 进入 execution(void org.apache.zookeeper.server.SyncRequestProcessor.processRequest(Request))
参数 sessionid:0x100064ba8b80000 type:create cxid:0x1 zxid:0x200000002 txntype:1 reqpath:n/a 参数结尾 时间:2023-09-05T19:45:23.973 进入 execution(boolean org.apache.zookeeper.server.ZKDatabase.append(Request))
参数 sessionid:0x100064ba8b80000 type:create cxid:0x1 zxid:0x200000002 txntype:1 reqpath:n/a 参数结尾 时间:2023-09-05T19:45:23.973 进入 execution(boolean org.apache.zookeeper.server.persistence.FileTxnSnapLog.append(Request))
参数 sessionid:0x100064ba8b80000 type:create cxid:0x1 zxid:0x200000002 txntype:1 reqpath:n/a 参数结尾 时间:2023-09-05T19:45:23.973 进入 execution(boolean org.apache.zookeeper.server.persistence.FileTxnLog.append(Request))
参数 sessionid:0x100064ba8b80000 type:create cxid:0x1 zxid:0x200000002 txntype:1 reqpath:n/a 参数结尾 时间:2023-09-05T19:45:23.976 进入 execution(void org.apache.zookeeper.server.quorum.SendAckRequestProcessor.processRequest(Request))
参数 sessionid:0x100064ba8b80000 type:create cxid:0x1 zxid:0x200000002 txntype:1 reqpath:n/a 参数结尾 时间:2023-09-05T19:45:23.976 进入 execution(void org.apache.zookeeper.server.quorum.CommitProcessor.commit(Request))
参数 sessionid:0x100064ba8b80000 type:create cxid:0x1 zxid:0x200000002 txntype:1 reqpath:n/a 参数结尾 14:45:23.976 [QuorumPeer[myid=2](plain=[0:0:0:0:0:0:0:0]:12181)(secure=disabled)] DEBUG org.apache.zookeeper.server.quorum.CommitProcessor - Committing request:: sessionid:0x100064ba8b80000 type:create cxid:0x1 zxid:0x200000002 txntype:1 reqpath:n/a
时间:2023-09-05T19:45:23.976 进入 execution(void org.apache.zookeeper.server.FinalRequestProcessor.processRequest(Request))
参数 sessionid:0x100064ba8b80000 type:create cxid:0x1 zxid:0x200000002 txntype:1 reqpath:n/a 参数结尾 14:45:23.976 [CommitProcessor:2] DEBUG org.apache.zookeeper.server.FinalRequestProcessor - Processing request:: sessionid:0x100064ba8b80000 type:create cxid:0x1 zxid:0x200000002 txntype:1 reqpath:n/a
时间:2023-09-05T19:45:23.976 进入 execution(DataTree.ProcessTxnResult org.apache.zookeeper.server.ZooKeeperServer.processTxn(Request))
参数 sessionid:0x100064ba8b80000 type:create cxid:0x1 zxid:0x200000002 txntype:1 reqpath:n/a 参数结尾 14:45:23.976 [CommitProcessor:2] DEBUG org.apache.zookeeper.common.PathTrie - data11
14:45:23.976 [CommitProcessor:2] DEBUG org.apache.zookeeper.server.DataTree - Digests are matching for Zxid: 200000002, Digest in log and actual tree: 4248210059
时间:2023-09-05T19:45:23.977 进入 execution(void org.apache.zookeeper.server.ZKDatabase.addCommittedProposal(Request))
参数 sessionid:0x100064ba8b80000 type:create cxid:0x1 zxid:0x200000002 txntype:1 reqpath:n/a 参数结尾
分析
步骤一:
将请求转发给leader
- 通过ZooKeeperServer.processPacket 获取客户端的请求然后
- 调用followerRequestProcessor.processRequest
- FollowerRequestProcessor处理,将请求提交给leader
- zks.getFollower().request(request); 请求提交给leader
- 然后使用CommitProcessor进行处理
步骤三:
处理leader的Proposal
- QuorumPeer线程运行执行follower.followLeader();
- 通过follower.processPacket读取到leader的提案
- Leader.PROPOSAL类型
- followerZooKeeperServer.logRequest添加到pendingTxns
- SyncRequestProcessor 将已提交日志同步到磁盘
- SendAckRequestProcessor 提交到磁盘后发送Ack日志
步骤五:
处理提交请求
- 与Proposal类似,使用 follower.processPacket处理
- Leader.COMMIT类型
- 处理pendingTxns的请求
- commitProcessor
- finalRequestProcessor
- 执行ZKDatabase.addCommittedProposal
node2
日志
bash
时间:2023-09-05T19:45:23.973 进入 execution(void org.apache.zookeeper.server.SyncRequestProcessor.processRequest(Request))
参数 sessionid:0x100064ba8b80000 type:create cxid:0x1 zxid:0x200000002 txntype:1 reqpath:n/a 参数结尾 时间:2023-09-05T19:45:23.973 进入 execution(boolean org.apache.zookeeper.server.ZKDatabase.append(Request))
参数 sessionid:0x100064ba8b80000 type:create cxid:0x1 zxid:0x200000002 txntype:1 reqpath:n/a 参数结尾 时间:2023-09-05T19:45:23.973 进入 execution(boolean org.apache.zookeeper.server.persistence.FileTxnSnapLog.append(Request))
参数 sessionid:0x100064ba8b80000 type:create cxid:0x1 zxid:0x200000002 txntype:1 reqpath:n/a 参数结尾 时间:2023-09-05T19:45:23.973 进入 execution(boolean org.apache.zookeeper.server.persistence.FileTxnLog.append(Request))
参数 sessionid:0x100064ba8b80000 type:create cxid:0x1 zxid:0x200000002 txntype:1 reqpath:n/a 参数结尾 时间:2023-09-05T19:45:23.976 进入 execution(void org.apache.zookeeper.server.quorum.SendAckRequestProcessor.processRequest(Request))
参数 sessionid:0x100064ba8b80000 type:create cxid:0x1 zxid:0x200000002 txntype:1 reqpath:n/a 参数结尾 时间:2023-09-05T19:45:23.976 进入 execution(void org.apache.zookeeper.server.quorum.CommitProcessor.commit(Request))
参数 sessionid:0x100064ba8b80000 type:create cxid:0x1 zxid:0x200000002 txntype:1 reqpath:n/a 参数结尾 14:45:23.976 [QuorumPeer[myid=2](plain=[0:0:0:0:0:0:0:0]:12181)(secure=disabled)] DEBUG org.apache.zookeeper.server.quorum.CommitProcessor - Committing request:: sessionid:0x100064ba8b80000 type:create cxid:0x1 zxid:0x200000002 txntype:1 reqpath:n/a
时间:2023-09-05T19:45:23.976 进入 execution(void org.apache.zookeeper.server.FinalRequestProcessor.processRequest(Request))
参数 sessionid:0x100064ba8b80000 type:create cxid:0x1 zxid:0x200000002 txntype:1 reqpath:n/a 参数结尾 14:45:23.976 [CommitProcessor:2] DEBUG org.apache.zookeeper.server.FinalRequestProcessor - Processing request:: sessionid:0x100064ba8b80000 type:create cxid:0x1 zxid:0x200000002 txntype:1 reqpath:n/a
时间:2023-09-05T19:45:23.976 进入 execution(DataTree.ProcessTxnResult org.apache.zookeeper.server.ZooKeeperServer.processTxn(Request))
参数 sessionid:0x100064ba8b80000 type:create cxid:0x1 zxid:0x200000002 txntype:1 reqpath:n/a 参数结尾 14:45:23.976 [CommitProcessor:2] DEBUG org.apache.zookeeper.common.PathTrie - data11
14:45:23.976 [CommitProcessor:2] DEBUG org.apache.zookeeper.server.DataTree - Digests are matching for Zxid: 200000002, Digest in log and actual tree: 4248210059
时间:2023-09-05T19:45:23.977 进入 execution(void org.apache.zookeeper.server.ZKDatabase.addCommittedProposal(Request))
参数 sessionid:0x100064ba8b80000 type:create cxid:0x1 zxid:0x200000002 txntype:1 reqpath:n/a 参数结尾
分析
步骤三:
处理leader的Proposal
- QuorumPeer线程运行执行follower.followLeader();
- 通过follower.processPacket读取到leader的提案
- Leader.PROPOSAL类型
- followerZooKeeperServer.logRequest添加到pendingTxns
- SyncRequestProcessor 将已提交日志同步到磁盘
- SendAckRequestProcessor 提交到磁盘后发送Ack日志
步骤五:
处理提交请求
- 与Proposal类似,使用 follower.processPacket处理
- Leader.COMMIT类型
- 处理pendingTxns的请求
- commitProcessor
- finalRequestProcessor
- 执行ZKDatabase.addCommittedProposal
node3
日志
css
时间:2023-09-05T19:45:23.960 进入 execution(void org.apache.zookeeper.server.quorum.Leader.submitLearnerRequest(Request))
参数 sessionid:0x100064ba8b80000 type:create cxid:0x1 zxid:0xfffffffffffffffe txntype:unknown reqpath:/data11 参数结尾 时间:2023-09-05T19:45:23.960 进入 execution(void org.apache.zookeeper.server.quorum.LeaderZooKeeperServer.submitLearnerRequest(Request))
参数 sessionid:0x100064ba8b80000 type:create cxid:0x1 zxid:0xfffffffffffffffe txntype:unknown reqpath:/data11 参数结尾 时间:2023-09-05T19:45:23.960 进入 execution(void org.apache.zookeeper.server.PrepRequestProcessor.processRequest(Request))
参数 sessionid:0x100064ba8b80000 type:create cxid:0x1 zxid:0xfffffffffffffffe txntype:unknown reqpath:/data11 参数结尾 14:45:23.963 [ProcessThread(sid:3 cport:-1):] DEBUG org.apache.zookeeper.server.SessionTrackerImpl - Checking session 0x100064ba8b80000
14:45:23.963 [ProcessThread(sid:3 cport:-1):] DEBUG org.apache.zookeeper.server.PrepRequestProcessor - Processing ACL: 31,s{'world,'anyone}
14:45:23.965 [ProcessThread(sid:3 cport:-1):] DEBUG org.apache.zookeeper.server.ZooKeeperServer - Permission requested: 4
14:45:23.965 [ProcessThread(sid:3 cport:-1):] DEBUG org.apache.zookeeper.server.ZooKeeperServer - ACLs for node: [31,s{'world,'anyone}
14:45:23.965 [ProcessThread(sid:3 cport:-1):] DEBUG org.apache.zookeeper.server.ZooKeeperServer - Client credentials: ['ip,'127.0.0.1
14:45:23.972 [ProcessThread(sid:3 cport:-1):] DEBUG org.apache.zookeeper.server.PrepRequestProcessor - Digest got from data tree is: 1371985504
14:45:23.972 [ProcessThread(sid:3 cport:-1):] DEBUG org.apache.zookeeper.server.PrepRequestProcessor - Digest got from outstandingChanges is: 1442819401
时间:2023-09-05T19:45:23.972 进入 execution(void org.apache.zookeeper.server.quorum.ProposalRequestProcessor.processRequest(Request))
参数 sessionid:0x100064ba8b80000 type:create cxid:0x1 zxid:0x200000002 txntype:1 reqpath:/data11 参数结尾 时间:2023-09-05T19:45:23.972 进入 execution(void org.apache.zookeeper.server.quorum.CommitProcessor.processRequest(Request))
参数 sessionid:0x100064ba8b80000 type:create cxid:0x1 zxid:0x200000002 txntype:1 reqpath:/data11 参数结尾 14:45:23.972 [ProcessThread(sid:3 cport:-1):] DEBUG org.apache.zookeeper.server.quorum.CommitProcessor - Processing request:: sessionid:0x100064ba8b80000 type:create cxid:0x1 zxid:0x200000002 txntype:1 reqpath:/data11
时间:2023-09-05T19:45:23.972 进入 execution(Leader.Proposal org.apache.zookeeper.server.quorum.Leader.propose(Request))
参数 sessionid:0x100064ba8b80000 type:create cxid:0x1 zxid:0x200000002 txntype:1 reqpath:/data11 参数结尾 14:45:23.972 [ProcessThread(sid:3 cport:-1):] DEBUG org.apache.zookeeper.server.quorum.Leader - Proposing:: sessionid:0x100064ba8b80000 type:create cxid:0x1 zxid:0x200000002 txntype:1 reqpath:/data11
时间:2023-09-05T19:45:23.972 进入 execution(void org.apache.zookeeper.server.SyncRequestProcessor.processRequest(Request))
参数 sessionid:0x100064ba8b80000 type:create cxid:0x1 zxid:0x200000002 txntype:1 reqpath:/data11 参数结尾 时间:2023-09-05T19:45:23.972 进入 execution(boolean org.apache.zookeeper.server.ZKDatabase.append(Request))
参数 sessionid:0x100064ba8b80000 type:create cxid:0x1 zxid:0x200000002 txntype:1 reqpath:/data11 参数结尾 时间:2023-09-05T19:45:23.972 进入 execution(boolean org.apache.zookeeper.server.persistence.FileTxnSnapLog.append(Request))
参数 sessionid:0x100064ba8b80000 type:create cxid:0x1 zxid:0x200000002 txntype:1 reqpath:/data11 参数结尾 时间:2023-09-05T19:45:23.972 进入 execution(boolean org.apache.zookeeper.server.persistence.FileTxnLog.append(Request))
参数 sessionid:0x100064ba8b80000 type:create cxid:0x1 zxid:0x200000002 txntype:1 reqpath:/data11 参数结尾 时间:2023-09-05T19:45:23.974 进入 execution(void org.apache.zookeeper.server.quorum.AckRequestProcessor.processRequest(Request))
参数 sessionid:0x100064ba8b80000 type:create cxid:0x1 zxid:0x200000002 txntype:1 reqpath:/data11 参数结尾 时间:2023-09-05T19:45:23.976 进入 execution(void org.apache.zookeeper.server.quorum.CommitProcessor.commit(Request))
参数 sessionid:0x100064ba8b80000 type:create cxid:0x1 zxid:0x200000002 txntype:1 reqpath:/data11 参数结尾 14:45:23.976 [LearnerHandler-/127.0.0.1:12551] DEBUG org.apache.zookeeper.server.quorum.CommitProcessor - Committing request:: sessionid:0x100064ba8b80000 type:create cxid:0x1 zxid:0x200000002 txntype:1 reqpath:/data11
时间:2023-09-05T19:45:23.976 进入 execution(void org.apache.zookeeper.server.quorum.Leader.ToBeAppliedRequestProcessor.processRequest(Request))
参数 14:45:23.976 [LearnerHandler-/127.0.0.1:12565] DEBUG org.apache.zookeeper.server.quorum.Leader - outstanding is 0
sessionid:0x100064ba8b80000 type:create cxid:0x1 zxid:0x200000002 txntype:1 reqpath:/data11 参数结尾 时间:2023-09-05T19:45:23.976 进入 execution(void org.apache.zookeeper.server.FinalRequestProcessor.processRequest(Request))
参数 sessionid:0x100064ba8b80000 type:create cxid:0x1 zxid:0x200000002 txntype:1 reqpath:/data11 参数结尾 14:45:23.976 [CommitProcessor:3] DEBUG org.apache.zookeeper.server.FinalRequestProcessor - Processing request:: sessionid:0x100064ba8b80000 type:create cxid:0x1 zxid:0x200000002 txntype:1 reqpath:/data11
时间:2023-09-05T19:45:23.976 进入 execution(DataTree.ProcessTxnResult org.apache.zookeeper.server.ZooKeeperServer.processTxn(Request))
参数 sessionid:0x100064ba8b80000 type:create cxid:0x1 zxid:0x200000002 txntype:1 reqpath:/data11 参数结尾 14:45:23.976 [CommitProcessor:3] DEBUG org.apache.zookeeper.common.PathTrie - data11
14:45:23.976 [CommitProcessor:3] DEBUG org.apache.zookeeper.server.DataTree - Digests are matching for Zxid: 200000002, Digest in log and actual tree: 4248210059
时间:2023-09-05T19:45:23.976 进入 execution(void org.apache.zookeeper.server.ZKDatabase.addCommittedProposal(Request))
参数 sessionid:0x100064ba8b80000 type:create cxid:0x1 zxid:0x200000002 txntype:1 reqpath:/data11 参数结尾
分析
步骤二:
leader提交Proposal
- 从follower 获取请求 Leader.submitLearnerRequest
- LearnerHandler 线程处理follower的请求
- 调用了ProposalRequestProcessor.processRequest
- ProposalRequestProcessor 使用的zxid进行提交数据
-
leader 通过 quorum.Leader.propose 提交提议
-
Leader.sendPacket 给发送提案包
javavoid sendPacket(QuorumPacket qp) { synchronized (forwardingFollowers) { for (LearnerHandler f : forwardingFollowers) { f.queuePacket(qp); } } }
-
调用syncRequestProcessor.processRequest
-
- SyncRequestProcessor 将已提交日志同步到磁盘
步骤四:
判断是否成功,发送commit请求
-
LearnerHandler 接收到follwers的Leader.ACK请求
- Leader.processAck
-
Leader.tryToCommit方法尝试提交,判断Proposal是否可以提交,核心逻辑大于半数Ack则提交
javapublic boolean hasAllQuorums() { for (QuorumVerifierAcksetPair qvAckset : qvAcksetPairs) { if (!qvAckset.getQuorumVerifier().containsQuorum(qvAckset.getAckset())) { return false; } } return true; } public boolean containsQuorum(Set<Long> ackSet) { return (ackSet.size() > half); }
-
发送Leader.COMMIT信息
步骤五:
处理提交请求
- zk.commitProcessor.commit(p.request);
- commitProcessor
- finalRequestProcessor
- 执行ZKDatabase.addCommittedProposal