一起学习ETCD系列——运维操作之etcdctl使用

文章目录

概要

本文主要用来总结ETCD客户端ctcdctl的命令操作，在运维过程中可能常常用到的。

一、命令

bash 复制代码

[root@test etcd-v3.5.11]# ./etcdctl -h
NAME:
        etcdctl - A simple command line client for etcd3.

USAGE:
        etcdctl [flags]

VERSION:
        3.5.11

API VERSION:
        3.5

COMMANDS:  #命令
        alarm disarm            Disarms all alarms  #关闭告警
        alarm list              Lists all alarms    #列出告警信息
        auth disable            Disables authentication #关闭认证
        auth enable             Enables authentication #开启认证
        auth status             Returns authentication status #查看认证状态
        check datascale         Check the memory usage of holding data for different workloads on a given server endpoint. #检查给定实例的内存使用情况
        check perf              Check the performance of the etcd cluster #检查etcd集群的性能
        compaction              Compacts the event history in etcd  #压缩etcd的历史事件
        defrag                  Defragments the storage of the etcd members with given endpoints #整理给定实例的存储碎片
        del                     Removes the specified key or range of keys [key, range_end) #删除key
        elect                   Observes and participates in leader election #加入leader选举
        endpoint hashkv         Prints the KV history hash for each endpoint in --endpoints  #打印给定若干实例KV history hash
        endpoint health         Checks the healthiness of endpoints specified in `--endpoints` flag #检查给定若干实例健康
        endpoint status         Prints out the status of endpoints specified in `--endpoints` flag  #打印给定若干实例状态
        get                     Gets the key or a range of keys #获取key值
        help                    Help about any command  #打印命令帮助信息
        lease grant             Creates leases  #创建一个租约
        lease keep-alive        Keeps leases alive (renew) #续约
        lease list              List all active leases  #列出所有租约
        lease revoke            Revokes leases   #撤销租约
        lease timetolive        Get lease information #获取租约详情，比如剩余有效期
        lock                    Acquires a named lock #获取命名锁
        make-mirror             Makes a mirror at the destination etcd cluster #指定一个etcd集群作为镜像集群
        member add              Adds a member into the cluster #添加一个成员到etcd集群
        member list             Lists all members in the cluster #列出etcd集群全部成员
        member promote          Promotes a non-voting member in the cluster #将成员从Follower状态到Candidate状态
        member remove           Removes a member from the cluster #移除成员
        member update           Updates a member in the cluster #更新成员
        move-leader             Transfers leadership to another etcd cluster member. #指定成员成为Leader状态
        put                     Puts the given key into the store #插入或更新key
        role add                Adds a new role #添加新角色
        role delete             Deletes a role  #删除角色
        role get                Gets detailed information of a role #获取角色的详细信息
        role grant-permission   Grants a key to a role #授予角色对key的权限，比如读写
        role list               Lists all roles  #列出全部的角色
        role revoke-permission  Revokes a key from a role #撤销角色对指定key的权限
        snapshot restore        Restores an etcd member snapshot to an etcd directory  #恢复快照到指定数据目录
        snapshot save           Stores an etcd node backend snapshot to a given file #保存快照到指定文件
        snapshot status         [deprecated] Gets backend snapshot status of a given file #[弃用] 获取给定快照文件的状态
        txn                     Txn processes all the requests in one transaction #开启事务
        user add                Adds a new user   #添加账户
        user delete             Deletes a user  #删除账户
        user get                Gets detailed information of a user #获取账户详情
        user grant-role         Grants a role to a user  #指定账户角色
        user list               Lists all users    #列出所有账户
        user passwd             Changes password of user  #修改账户密码
        user revoke-role        Revokes a role from a user #撤销账户角色
        version                 Prints the version of etcdctl  #打印ectd当前版本
        watch                   Watches events stream on keys or prefixes #监听一些key或共同前缀key的事件【】

OPTIONS:  #命令的可选参数
      --cacert=""                               verify certificates of TLS-enabled secure servers using this CA bundle #服务端使用https时，使用ca文件进行验证
      --cert=""                                 identify secure client using this TLS certificate file #指定HTTPS的TLS证书文件
      --command-timeout=5s                      timeout for short running command (excluding dial timeout) #命令执行超时时间
      --debug[=false]                           enable client-side debug logging #启用客户端调试日志，默认false
      --dial-timeout=2s                         dial timeout for client connections #指定连接超时时间
  -d, --discovery-srv=""                        domain name to query for SRV records describing cluster endpoints #用于查询描述集群端点的srv记录的域名
      --discovery-srv-name=""                   service name to query when using DNS discovery #使用DNS发现时要查询的服务名称
      --endpoints=[127.0.0.1:2379]              gRPC endpoints #指定实例访问ip+端口
  -h, --help[=false]                            help for etcdctl
      --hex[=false]                             print byte strings as hex encoded strings #将字节字符串打印为十六进制编码字符串
      --insecure-discovery[=true]               accept insecure SRV records describing cluster endpoints #接受描述集群端点的不安全的srv记录
      --insecure-skip-tls-verify[=false]        skip server certificate verification (CAUTION: this option should be enabled only for testing purposes)#跳过服务器证书验证（注意：此选项应仅用于测试目的）
      --insecure-transport[=true]               disable transport security for client connections #客户端禁用安全传输
      --keepalive-time=2s                       keepalive time for client connections #客户端连接的keepalive时间
      --keepalive-timeout=6s                    keepalive timeout for client connections #客户端连接的keepalive超时时间
      --key=""                                  identify secure client using this TLS key file #使用此 TLS 密钥文件识别安全客户端
      --password=""                             password for authentication (if this option is used, --user option shouldn't include password) #用于身份验证的密码（如果使用此选项，--user 选项不应包含密码）
      --user=""                                 username[:password] for authentication (prompt if password is not supplied)#用于身份验证（如果未提供密码则提示）
  -w, --write-out="simple"                      set the output format (fields, json, protobuf, simple, table) #输出格式,默认simple，个人感觉table格式看着最舒服

二、实操

etcdctl [command] -h 可以查看具体命令的使用说明：

bash 复制代码

[root@test etcd-v3.5.11]# ./etcdctl lease -h
NAME:
        lease - Lease related commands
USAGE:
        etcdctl lease <subcommand> [flags]
API VERSION:
        3.5
COMMANDS:
        grant           Creates leases
        keep-alive      Keeps leases alive (renew)
        list            List all active leases
        revoke          Revokes leases
        timetolive      Get lease information
OPTIONS:
  -h, --help[=false]    help for lease

[root@test etcd-v3.5.11]# ./etcdctl lease grant -h
NAME:
        lease grant - Creates leases
USAGE:
        etcdctl lease grant <ttl> [flags]  #详细使用说明
OPTIONS:
  -h, --help[=false]    help for grant

2.1、基本操作

bash 复制代码

#插入key
[root@test etcd-v3.5.11]# ./etcdctl put /test/key1 value1
OK
#创建500s的租约，其id为43e58d2bb5a3102d
[root@test etcd-v3.5.11]# ./etcdctl lease grant 500
lease 43e58d2bb5a3102d granted with TTL(500s)
#获取租约详情
[root@test etcd-v3.5.11]# ./etcdctl lease timetolive 43e58d2bb5a3102d
lease 43e58d2bb5a3102d granted with TTL(500s), remaining(470s)
#插入key，并指定租约
[root@test etcd-v3.5.11]# ./etcdctl put /test/key2 value2 --lease=43e58d2bb5a3102d
OK
#获取
[root@test etcd-v3.5.11]# ./etcdctl get /test/key1
/test/key1
value1
[root@test etcd-v3.5.11]# ./etcdctl get /test/key2
/test/key2
value2
#删除
[root@test etcd-v3.5.11]# ./etcdctl del /test/key2
1
[root@test etcd-v3.5.11]# ./etcdctl get /test/key1 -w josn
{"header":{"cluster_id":11073961802084797819,"member_id":8518302759694844901,"revision":10769,"raft_term":15},"kvs":[{"key":"L3Rlc3Qva2V5MQ==","create_revision":10759,"mod_revision":10761,"version":3,"value":"dmFsdWUxNA=="}],"count":1}
#cluster_id：请求的etcd集群ID。
#member_id：请求的etcd节点ID。
#revision：etcd服务端当前全局数据版本号。对任一key的put或delete操作都会使revision自增1。revision=1是etcd的保留版本号，因此用户的key版本号将从2开始。
#raft_term：etcd当前raft主节点任期号。
#create_revision：当前key创建时全局数据版本号revision的值。
#mod_revision：当前key最后一次修改时全局数据版本号revision的值。
#version：当前key的数据版本号。key创建时version为1，对当前key进行put操作会使version自增1，将key删除后，重新创建，version又会从1开始计数。

2.2、watch

监听key，可以看到key的事件

bash 复制代码

[root@test etcd-v3.5.11]# ./etcdctl watch /test/key3
PUT
/test/key3
value3
PUT
/test/key3
value31
PUT
/test/key3
value33
DELETE
/test/key3
PUT
/test/key3
value35

操作key

bash 复制代码

[root@test etcd-v3.5.11]# ./etcdctl put /test/key3 value3
OK
[root@test etcd-v3.5.11]# ./etcdctl put /test/key3 value31
OK
[root@test etcd-v3.5.11]# ./etcdctl put /test/key3 value33
OK
[root@test etcd-v3.5.11]# ./etcdctl del /test/key3
1
[root@test etcd-v3.5.11]# ./etcdctl put /test/key3 value35
OK

2.3、租约

bash 复制代码

#创建500s的租约，其id为43e58d2bb5a3102d
[root@test etcd-v3.5.11]# ./etcdctl lease grant 500
lease 43e58d2bb5a3102d granted with TTL(500s)
#获取租约详情
[root@test etcd-v3.5.11]# ./etcdctl lease timetolive 43e58d2bb5a3102d
lease 43e58d2bb5a3102d granted with TTL(500s), remaining(470s)
#只续约一次
[root@test etcd-v3.5.11]# ./etcdctl lease keep-alive --once=true 43e58d2bb5a3102d
#一直自动续约
[root@test etcd-v3.5.11]# ./etcdctl lease keep-alive 43e58d2bb5a3102d
#删除租约
[root@test etcd-v3.5.11]# ./etcdctl lease revoke 43e58d2bb5a3102d

2.4、分布式锁

etcd中分布式锁的结构是：lock_key/lease_id。

实现过程（源码）如下：

准备
客户端连接 Etcd，以 /lock/mylock 为前缀创建全局唯一的 key，假设第一个客户端对应的 key="/lock/mylock/UUID1"，第二个为 key="/lock/mylock/UUID2"；客户端分别为自己的 key 创建租约 - Lease，租约的长度根据业务耗时确定，假设为 15s；
创建租约
当一个客户端持有锁期间，其它客户端只能等待，为了避免等待期间租约失效，客户端需创建一个定时任务作为"心跳"进行续约。此外，如果持有锁期间客户端崩溃，心跳停止，key 将因租约到期而被删除，从而锁释放，避免死锁。
客户端写入自己全局唯一的 key
通过 PUT 操作，将步骤 1 中创建的 key 绑定租约写入 Etcd，根据 Etcd 的 Revision 机制，假设两个客户端 put 操作返回的 Revision 分别为 1、2，客户端需记录 Revision 用以接下来判断自己是否获得锁。
客户端判断是否获得锁
客户端以前缀 /lock/mylock 读取 keyValue 列表（keyValue 中带有 key 对应的 Revision），判断自己 key 的 Revision 是否为当前列表中最小的，如果是则认为获得锁；否则监听列表中前一个 Revision 比自己小的 key 的删除事件，一旦监听到删除事件或者因租约失效而删除的事件，则自己获得锁。
执行业务
获得锁后，操作共享资源，执行业务代码。
释放锁
完成业务流程后，释放锁,通过删除对应的key完成。

测试：

开三个窗口

bash 复制代码

#获取锁
[root@test etcd-v3.5.11]# ./etcdctl lock /lock/test --ttl=5
/lock/test/43e58d437ddbc834

bash 复制代码

#等待锁
[root@test etcd-v3.5.11]# ./etcdctl lock /lock/test --ttl=5

bash 复制代码

#watch监控/lock/test为前缀的key变化
[root@kdzl etcd-v3.5.11]# ./etcdctl watch /lock/test --prefix
PUT
/lock/test/43e58d437ddbc834
PUT
/lock/test/43e58d437ddbc838

2.5、角色

bash 复制代码

[root@test etcd-v3.5.11]# ./etcdctl role add book
Role book created
[root@test etcd-v3.5.11]# ./etcdctl role add man
Role man created
[root@test etcd-v3.5.11]# ./etcdctl role list
book
man
[root@test etcd-v3.5.11]# ./etcdctl role get man
Role man
KV Read:
KV Write:
#给key授权角色
#etcdctl role grant-permission [options] <role name> <permission type> <key> [endkey] [flags]
[root@test etcd-v3.5.11]# ./etcdctl role grant-permission man read /test
Role man updated
[root@test etcd-v3.5.11]# ./etcdctl role get man
Role man
KV Read:
        /test
KV Write:
#移除key角色
[root@test etcd-v3.5.11]# ./etcdctl role revoke-permission man /test
Permission of key /test is revoked from role man
[root@test etcd-v3.5.11]# ./etcdctl role get man
Role man
KV Read:
KV Write:

2.6、用户

bash 复制代码

#创建用户
[root@test etcd-v3.5.11]# ./etcdctl user add root:root
User root created
[root@test etcd-v3.5.11]# ./etcdctl user add test --new-user-password=12345
User test created
[root@test etcd-v3.5.11]# ./etcdctl user add pick:25823
User pick created
#列出全部的用户
[root@test etcd-v3.5.11]# ./etcdctl user list
root
test
[root@test etcd-v3.5.11]# ./etcdctl user get test
User: test
Roles:
#给账号一个角色,一个账户可以有多个角色
[root@test etcd-v3.5.11]# ./etcdctl user grant-role test man
Role man is granted to user test
[root@test etcd-v3.5.11]# ./etcdctl user get test
User: test
Roles: man
#删除角色
[root@test etcd-v3.5.11]# ./etcdctl user del pick
User pick deleted
#撤销用户角色
[root@test etcd-v3.5.11]# ./etcdctl user revoke-role test man
Role man is revoked from user test

2.7、认证

启用auth需要一个root用户，并且root用户有root角色

bash 复制代码

[root@test etcd-v3.5.11]# ./etcdctl role add root
Role root created
[root@test etcd-v3.5.11]# ./etcdctl role grant-permission --prefix=true root readwrite /
Role root updated
[root@test etcd-v3.5.11]# ./etcdctl user grant-role root root
Role root is granted to user root
[root@test etcd-v3.5.11]# ./etcdctl auth enable
Authentication Enabled
[root@test etcd-v3.5.11]# ./etcdctl auth status --user=root:root
Authentication Status: true
AuthRevision: 15

换成 test 用户操作

bash 复制代码

[root@test etcd-v3.5.11]# ./etcdctl role grant-permission --prefix=true man read /test --user=root:root  #对/test前缀的key有只读权限
Role man updated
[root@test etcd-v3.5.11]# ./etcdctl role get man --user=root:root
Role man
KV Read:
        /test
KV Write:
[root@test etcd-v3.5.11]# ./etcdctl user grant-role test man --user=root:root
Role man is granted to user test
[root@test etcd-v3.5.11]# ./etcdctl get /test/key1 --user=test:12345
/test/key1
value14
[root@test etcd-v3.5.11]# ./etcdctl put /test/key5 value5 --user=test:12345
{"level":"warn","ts":"2024-01-25T23:38:54.753123+0800","logger":"etcd-client","caller":"v3@v3.5.11/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000362e00/127.0.0.1:2379","attempt":0,"error":"rpc error: code = PermissionDenied desc = etcdserver: permission denied"}
Error: etcdserver: permission denied

可以看到权限生效

2.8、集群

1：查看每个实例状态,即检查etcd集群状态

bash 复制代码

[root@test etcd-v3.5.11]# ./etcdctl endpoint status -w table --endpoints"=127.0.0.1:2379,127.0.0.1:2369,127.0.0.1:2359"
+----------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|    ENDPOINT    |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+----------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| 127.0.0.1:2379 | 7637173a60d743e5 |  3.5.11 |   21 MB |      true |      false |        15 |      10923 |              10923 |        |
| 127.0.0.1:2369 | ce6b0efed2935560 |  3.5.11 |   21 MB |     false |      false |        15 |      10923 |              10923 |        |
| 127.0.0.1:2359 | 6ea1523f71bf92f0 |  3.5.11 |   21 MB |     false |      false |        15 |      10923 |              10923 |        |
+----------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+

2:查看健康状态

bash 复制代码

#TOOK表示心跳
[root@test etcd-v3.5.11]# ./etcdctl endpoint health --endpoints="127.0.0.1:2379,127.0.0.1:2369,127.0.0.1:2359" -w table
+----------------+--------+-------------+-------+
|    ENDPOINT    | HEALTH |    TOOK     | ERROR |
+----------------+--------+-------------+-------+
| 127.0.0.1:2369 |   true | 16.178532ms |       |
| 127.0.0.1:2379 |   true | 17.826437ms |       |
| 127.0.0.1:2359 |   true | 19.841588ms |       |
+----------------+--------+-------------+-------+
[root@test etcd-v3.5.11]# ./etcdctl endpoint hashkv --endpoints="127.0.0.1:2379,127.0.0.1:2369,127.0.0.1:2359" -w table
+----------------+------------+
|    ENDPOINT    |    HASH    |
+----------------+------------+
| 127.0.0.1:2379 | 2618695019 |
| 127.0.0.1:2369 | 2618695019 |
| 127.0.0.1:2359 | 2618695019 |
+----------------+------------+

3：切换集群leader

原来集群leader实例是127.0.0.1:2379,现在通过move-leader命令将leader切换成实例127.0.0.1:2359

bash 复制代码

[root@test etcd-v3.5.11]# ./etcdctl move-leader 6ea1523f71bf92f0 --endpoints="127.0.0.1:2379,127.0.0.1:2369,127.0.0.1:2359"
Leadership transferred from 7637173a60d743e5 to 6ea1523f71bf92f0
[root@test etcd-v3.5.11]# ./etcdctl endpoint status --endpoints="127.0.0.1:2379,127.0.0.1:2369,127.0.0.1:2359" -w table
+----------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|    ENDPOINT    |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+----------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| 127.0.0.1:2379 | 7637173a60d743e5 |  3.5.11 |   21 MB |     false |      false |        19 |      10933 |              10933 |        |
| 127.0.0.1:2369 | ce6b0efed2935560 |  3.5.11 |   21 MB |     false |      false |        19 |      10933 |              10933 |        |
| 127.0.0.1:2359 | 6ea1523f71bf92f0 |  3.5.11 |   21 MB |      true |      false |        19 |      10933 |              10933 |        |
+----------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+

4：查看集群成员

bash 复制代码

[root@test etcd-v3.5.11]# ./etcdctl  member list -w table
+------------------+---------+--------+-----------------------+-----------------------+------------+
|        ID        | STATUS  |  NAME  |      PEER ADDRS       |     CLIENT ADDRS      | IS LEARNER |
+------------------+---------+--------+-----------------------+-----------------------+------------+
| 6ea1523f71bf92f0 | started | etcd3 | http://127.0.0.1:2360 | http://127.0.0.1:2359 |      false |
| 7637173a60d743e5 | started | etcd1 | http://127.0.0.1:2380 | http://127.0.0.1:2379 |      false |
| ce6b0efed2935560 | started | etcd2 | http://127.0.0.1:2370 | http://127.0.0.1:2369 |      false |
+------------------+---------+--------+-----------------------+-----------------------+------------+

5：添加成员（一个learner成员，不参与投票）

1）etcdctl member add 命令将新成员添加到集群

bash 复制代码

[root@testl etcd-v3.5.11]# ./etcdctl  member add  etcd04 --peer-urls="http://127.0.0.16:2350" --learner=true
Member 690b31e00158f43a added to cluster 99ae9d962d824d7b

ETCD_NAME="etcd4"
ETCD_INITIAL_CLUSTER="etcd4=http://127.0.0.1:2350,etcd3=http://127.0.0.1:2360,etcd1=http://127.0.0.1:2380,etcd2=http://127.0.0.1:2370"
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://127.0.0.1:2350"
ETCD_INITIAL_CLUSTER_STATE="existing"
#可以看到该成员尚未启动
[root@test etcd-v3.5.11]# ./etcdctl  member list -w table
+------------------+-----------+--------+----------------------------+-----------------------+------------+
|        ID        |  STATUS   |  NAME  |         PEER ADDRS         |     CLIENT ADDRS      | IS LEARNER |
+------------------+-----------+--------+----------------------------+-----------------------+------------+
| 690b31e00158f43a | unstarted |       | http://127.0.0.1:2350 |                       |       true |
| 6ea1523f71bf92f0 |   started | etcd3 |      http://127.0.0.1:2360 | http://127.0.0.1:2359 |      false |
| 7637173a60d743e5 |   started | etcd1 |      http://127.0.0.1:2380 | http://127.0.0.1:2379 |      false |
| ce6b0efed2935560 |   started | etcd2 |      http://127.0.0.1:2370 | http://127.0.0.1:2369 |      false |
+------------------+-----------+--------+----------------------------+-----------------------+------------+

2）启动新成员
回溯集群安装篇

配置文件如下：

bash 复制代码

mkdir -p /usr/local/etcd-v3.5.11/etcd4/data
vim /usr/local/etcd-v3.5.11/etcd4/conf.yml

yaml 复制代码

name: etcd4
data-dir: /usr/local/etcd-v3.5.11/etcd4/data
initial-advertise-peer-urls: http://127.0.0.1:2350
listen-peer-urls: http://127.0.0.1:2350
listen-client-urls: http://172.20.101.222:2349,http://127.0.0.1:2349
advertise-client-urls: http://127.0.0.1:2349
initial-cluster-token: test-etcd-cluster
initial-cluster: etcd1=http://127.0.0.1:2380,etcd2=http://127.0.0.1:2370,etcd3=http://127.0.0.1:2360,etcd4=http://127.0.0.1:2350
initial-cluster-state: existing   #注意，这里一定要标记为existing

bash 复制代码

/usr/local/etcd-v3.5.11/etcd --config-file /usr/local/etcd-v3.5.11/etcd4/conf.yml

可以看到已经成功加入集群

bash 复制代码

[root@test etcd-v3.5.11]# ./etcdctl  member list -w table
+------------------+---------+--------+-----------------------+-----------------------+------------+
|        ID        | STATUS  |  NAME  |      PEER ADDRS       |     CLIENT ADDRS      | IS LEARNER |
+------------------+---------+--------+-----------------------+-----------------------+------------+
| 6ea1523f71bf92f0 | started | etcd3 | http://127.0.0.1:2360 | http://127.0.0.1:2359 |      false |
| 7637173a60d743e5 | started | etcd1 | http://127.0.0.1:2380 | http://127.0.0.1:2379 |      false |
| c64a0c36c4a5869f | started | etcd4 | http://127.0.0.1:2350 | http://127.0.0.1:2349 |       true |
| ce6b0efed2935560 | started | etcd2 | http://127.0.0.1:2370 | http://127.0.0.1:2369 |      false |
+------------------+---------+--------+-----------------------+-----------------------+------------+

6：learner成员提升为follower成员

实例http://127.0.0.1:2350还是learner，没有投票权，用member promote命令提升下

bash 复制代码

[root@test etcd-v3.5.11]# ./etcdctl  member  promote c64a0c36c4a5869f
Member c64a0c36c4a5869f promoted in cluster 99ae9d962d824d7b
[root@test etcd-v3.5.11]# ./etcdctl  member list -w table
+------------------+---------+--------+-----------------------+-----------------------+------------+
|        ID        | STATUS  |  NAME  |      PEER ADDRS       |     CLIENT ADDRS      | IS LEARNER |
+------------------+---------+--------+-----------------------+-----------------------+------------+
| 6ea1523f71bf92f0 | started | etcd3 | http://127.0.0.1:2360 | http://127.0.0.1:2359 |      false |
| 7637173a60d743e5 | started | etcd1 | http://127.0.0.1:2380 | http://127.0.0.1:2379 |      false |
| c64a0c36c4a5869f | started | etcd4 | http://127.0.0.1:2350 | http://127.0.0.1:2349 |      false |  #可以看到已经不是learner了
| ce6b0efed2935560 | started | etcd2 | http://127.0.0.1:2370 | http://127.0.0.1:2369 |      false |
+------------------+---------+--------+-----------------------+-----------------------+------------+