前言
本文的问题 主要是 衍生自spring-boot-acurator 定时检测 redis 集群导致 "IOException: Too many open files"
对于这里 pipe 的使用, 也是很神奇, 因为 貌似没有用过这类 api
然后 这里调研一下, 然后 追根究底到最终, 是到了 jdk 的 c 代码的调用, 创建的 pipe
然后 大概看了一下 java 的 PipeReader/Writer, 发现 其实没有到 linux 的 pipe, 仅仅是 java language level 实现的一个管道读写
这部分在 java 语言这一层 是无感知的, 只能使用fd
这里来看一下 使用了 spring-data-redis 的项目中的那些 pipe, anon_inode 相关的 FileDescriptor
然后 这部分 FileDescriptor 较难找到
这里 来探究一下 这部分的 FileDescriptor 创建的地方
pipe 部分的 FileDescriptor 如下, 合计 102 项
root@ubuntu:~/docker/HelloWorld# ll /proc/58983/fd | grep pipe
lr-x------ 1 root root 64 Jun 26 04:12 36 -> pipe:[6215597]
l-wx------ 1 root root 64 Jun 26 04:12 37 -> pipe:[6215597]
lr-x------ 1 root root 64 Jun 26 04:12 39 -> pipe:[6215598]
// 省略部分 fd 的信息
lr-x------ 1 root root 64 Jun 26 04:12 201 -> pipe:[6216982]
l-wx------ 1 root root 64 Jun 26 04:12 202 -> pipe:[6216982]
lr-x------ 1 root root 64 Jun 26 04:12 204 -> pipe:[6216983]
l-wx------ 1 root root 64 Jun 26 04:12 205 -> pipe:[6216983]
anon_inode 部分的 FileDescriptor 如下, 合计 51 项
root@ubuntu:~/docker/HelloWorld# ll /proc/58983/fd | grep ano
lrwx------ 1 root root 64 Jun 26 04:12 38 -> anon_inode:[eventpoll]
lrwx------ 1 root root 64 Jun 26 04:12 41 -> anon_inode:[eventpoll]
lrwx------ 1 root root 64 Jun 26 04:12 44 -> anon_inode:[eventpoll]
// 省略部分 fd 的信息
lrwx------ 1 root root 64 Jun 26 04:12 200 -> anon_inode:[eventpoll]
lrwx------ 1 root root 64 Jun 26 04:12 203 -> anon_inode:[eventpoll]
lrwx------ 1 root root 64 Jun 26 04:12 206 -> anon_inode:[eventpoll]
p ipe 和 anon_inode
创建 pipe连接 和 anon_inode连接 的地方是在这里
当前 fd 列表信息如下
root@ubuntu:~/docker/HelloWorld# ll /proc/59215/fd
total 0
dr-x------ 2 root root 0 Jun 26 04:26 ./
dr-xr-xr-x 9 root root 0 Jun 26 04:25 ../
lr-x------ 1 root root 64 Jun 26 04:26 0 -> /dev/null
l-wx------ 1 root root 64 Jun 26 04:26 1 -> /root/docker/HelloWorld/logs/nohup.log
l-wx------ 1 root root 64 Jun 26 04:26 10 -> /root/docker/HelloWorld/logs/error-2023-06-26.0.log
lr-x------ 1 root root 64 Jun 26 04:26 11 -> /usr/local/ProgramFiles/jdk1.8.0_291/jre/lib/ext/jfxrt.jar
l-wx------ 1 root root 64 Jun 26 04:26 12 -> /root/logs/nacos/config.log
lrwx------ 1 root root 64 Jun 26 04:26 13 -> socket:[6223806]
l-wx------ 1 root root 64 Jun 26 04:26 14 -> /root/logs/nacos/naming.log
lr-x------ 1 root root 64 Jun 26 04:26 15 -> /usr/local/ProgramFiles/jdk1.8.0_291/jre/lib/jsse.jar
lr-x------ 1 root root 64 Jun 26 04:26 16 -> /dev/random
lr-x------ 1 root root 64 Jun 26 04:26 17 -> /dev/urandom
lr-x------ 1 root root 64 Jun 26 04:26 18 -> /dev/random
lr-x------ 1 root root 64 Jun 26 04:26 19 -> /dev/random
l-wx------ 1 root root 64 Jun 26 04:26 2 -> /root/docker/HelloWorld/logs/nohup.log
lr-x------ 1 root root 64 Jun 26 04:26 20 -> /dev/urandom
lr-x------ 1 root root 64 Jun 26 04:26 21 -> /dev/urandom
lr-x------ 1 root root 64 Jun 26 04:26 22 -> /usr/local/ProgramFiles/jdk1.8.0_291/jre/lib/ext/cldrdata.jar
lr-x------ 1 root root 64 Jun 26 04:26 23 -> /usr/local/ProgramFiles/jdk1.8.0_291/jre/lib/ext/localedata.jar
lrwx------ 1 root root 64 Jun 26 04:26 24 -> socket:[6226176]
lrwx------ 1 root root 64 Jun 26 04:26 25 -> socket:[6226178]
lr-x------ 1 root root 64 Jun 26 04:26 26 -> /usr/local/ProgramFiles/jdk1.8.0_291/jre/lib/jce.jar
lr-x------ 1 root root 64 Jun 26 04:26 27 -> /usr/local/ProgramFiles/jdk1.8.0_291/jre/lib/ext/sunpkcs11.jar
lr-x------ 1 root root 64 Jun 26 04:26 28 -> /usr/local/ProgramFiles/jdk1.8.0_291/jre/lib/ext/sunec.jar
lr-x------ 1 root root 64 Jun 26 04:26 29 -> /usr/local/ProgramFiles/jdk1.8.0_291/jre/lib/ext/sunjce_provider.jar
lr-x------ 1 root root 64 Jun 26 04:26 3 -> /usr/local/ProgramFiles/jdk1.8.0_291/jre/lib/rt.jar
lrwx------ 1 root root 64 Jun 26 04:26 30 -> socket:[6223845]
lr-x------ 1 root root 64 Jun 26 04:26 31 -> /usr/local/ProgramFiles/jdk1.8.0_291/jre/lib/resources.jar
lrwx------ 1 root root 64 Jun 26 04:26 32 -> socket:[6223868]
lrwx------ 1 root root 64 Jun 26 04:26 33 -> socket:[6223871]
lr-x------ 1 root root 64 Jun 26 04:26 34 -> /usr/local/ProgramFiles/jdk1.8.0_291/jre/lib/charsets.jar
l-wx------ 1 root root 64 Jun 26 04:26 4 -> /root/docker/HelloWorld/logs/debug-2023-06-26.0.log
lrwx------ 1 root root 64 Jun 26 04:26 5 -> socket:[6223797]
lr-x------ 1 root root 64 Jun 26 04:26 6 -> /root/docker/HelloWorld/HelloWorld.jar
lr-x------ 1 root root 64 Jun 26 04:26 7 -> /root/docker/HelloWorld/HelloWorld.jar
l-wx------ 1 root root 64 Jun 26 04:26 8 -> /root/docker/HelloWorld/logs/info-2023-06-26.0.log
l-wx------ 1 root root 64 Jun 26 04:26 9 -> /root/docker/HelloWorld/logs/warn-2023-06-26.0.log
单步调试过了 provider.openSelector 之后, fd 列表如下
可以看到除了增加了两个 pipe 的 fd 和一个 anon_inode 之外还增加了一部分 provider.openSelector 期间需要加载的 jar, 连接 等等
root@ubuntu:~/docker/HelloWorld# ll /proc/59215/fd
total 0
dr-x------ 2 root root 0 Jun 26 04:26 ./
dr-xr-xr-x 9 root root 0 Jun 26 04:25 ../
lr-x------ 1 root root 64 Jun 26 04:26 0 -> /dev/null
l-wx------ 1 root root 64 Jun 26 04:26 1 -> /root/docker/HelloWorld/logs/nohup.log
l-wx------ 1 root root 64 Jun 26 04:26 10 -> /root/docker/HelloWorld/logs/error-2023-06-26.0.log
lr-x------ 1 root root 64 Jun 26 04:26 11 -> /usr/local/ProgramFiles/jdk1.8.0_291/jre/lib/ext/jfxrt.jar
l-wx------ 1 root root 64 Jun 26 04:26 12 -> /root/logs/nacos/config.log
lrwx------ 1 root root 64 Jun 26 04:26 13 -> socket:[6223806]
l-wx------ 1 root root 64 Jun 26 04:26 14 -> /root/logs/nacos/naming.log
lr-x------ 1 root root 64 Jun 26 04:26 15 -> /usr/local/ProgramFiles/jdk1.8.0_291/jre/lib/jsse.jar
lr-x------ 1 root root 64 Jun 26 04:26 16 -> /dev/random
lr-x------ 1 root root 64 Jun 26 04:26 17 -> /dev/urandom
lr-x------ 1 root root 64 Jun 26 04:26 18 -> /dev/random
lr-x------ 1 root root 64 Jun 26 04:26 19 -> /dev/random
l-wx------ 1 root root 64 Jun 26 04:26 2 -> /root/docker/HelloWorld/logs/nohup.log
lr-x------ 1 root root 64 Jun 26 04:26 20 -> /dev/urandom
lr-x------ 1 root root 64 Jun 26 04:26 21 -> /dev/urandom
lr-x------ 1 root root 64 Jun 26 04:26 22 -> /usr/local/ProgramFiles/jdk1.8.0_291/jre/lib/ext/cldrdata.jar
lr-x------ 1 root root 64 Jun 26 04:26 23 -> /usr/local/ProgramFiles/jdk1.8.0_291/jre/lib/ext/localedata.jar
lrwx------ 1 root root 64 Jun 26 04:26 24 -> socket:[6226176]
lr-x------ 1 root root 64 Jun 26 04:26 25 -> pipe:[6230147]
lr-x------ 1 root root 64 Jun 26 04:26 26 -> /usr/local/ProgramFiles/jdk1.8.0_291/jre/lib/jce.jar
lr-x------ 1 root root 64 Jun 26 04:26 27 -> /usr/local/ProgramFiles/jdk1.8.0_291/jre/lib/ext/sunpkcs11.jar
lr-x------ 1 root root 64 Jun 26 04:26 28 -> /usr/local/ProgramFiles/jdk1.8.0_291/jre/lib/ext/sunec.jar
lr-x------ 1 root root 64 Jun 26 04:26 29 -> /usr/local/ProgramFiles/jdk1.8.0_291/jre/lib/ext/sunjce_provider.jar
lr-x------ 1 root root 64 Jun 26 04:26 3 -> /usr/local/ProgramFiles/jdk1.8.0_291/jre/lib/rt.jar
lrwx------ 1 root root 64 Jun 26 04:26 30 -> socket:[6223845]
lr-x------ 1 root root 64 Jun 26 04:26 31 -> /usr/local/ProgramFiles/jdk1.8.0_291/jre/lib/resources.jar
lrwx------ 1 root root 64 Jun 26 04:26 32 -> socket:[6223868]
lrwx------ 1 root root 64 Jun 26 04:26 33 -> socket:[6223871]
lr-x------ 1 root root 64 Jun 26 04:26 34 -> /usr/local/ProgramFiles/jdk1.8.0_291/jre/lib/charsets.jar
l-wx------ 1 root root 64 Jun 26 04:36 35 -> pipe:[6230147]
lrwx------ 1 root root 64 Jun 26 04:36 36 -> anon_inode:[eventpoll]
l-wx------ 1 root root 64 Jun 26 04:26 4 -> /root/docker/HelloWorld/logs/debug-2023-06-26.0.log
lrwx------ 1 root root 64 Jun 26 04:26 5 -> socket:[6223797]
lr-x------ 1 root root 64 Jun 26 04:26 6 -> /root/docker/HelloWorld/HelloWorld.jar
lr-x------ 1 root root 64 Jun 26 04:26 7 -> /root/docker/HelloWorld/HelloWorld.jar
l-wx------ 1 root root 64 Jun 26 04:26 8 -> /root/docker/HelloWorld/logs/info-2023-06-26.0.log
l-wx------ 1 root root 64 Jun 26 04:26 9 -> /root/docker/HelloWorld/logs/warn-2023-06-26.0.log
后面的每一次 调用, 就仅仅是增加了 一对 pipe 的 fd 和 一个 anon_inode 的 fd 了
root@ubuntu:~/docker/HelloWorld# ll /proc/59215/fd
total 0
dr-x------ 2 root root 0 Jun 26 04:26 ./
dr-xr-xr-x 9 root root 0 Jun 26 04:25 ../
lr-x------ 1 root root 64 Jun 26 04:26 0 -> /dev/null
l-wx------ 1 root root 64 Jun 26 04:26 1 -> /root/docker/HelloWorld/logs/nohup.log
l-wx------ 1 root root 64 Jun 26 04:26 10 -> /root/docker/HelloWorld/logs/error-2023-06-26.0.log
lr-x------ 1 root root 64 Jun 26 04:26 11 -> /usr/local/ProgramFiles/jdk1.8.0_291/jre/lib/ext/jfxrt.jar
l-wx------ 1 root root 64 Jun 26 04:26 12 -> /root/logs/nacos/config.log
lrwx------ 1 root root 64 Jun 26 04:26 13 -> socket:[6223806]
l-wx------ 1 root root 64 Jun 26 04:26 14 -> /root/logs/nacos/naming.log
lr-x------ 1 root root 64 Jun 26 04:26 15 -> /usr/local/ProgramFiles/jdk1.8.0_291/jre/lib/jsse.jar
lr-x------ 1 root root 64 Jun 26 04:26 16 -> /dev/random
lr-x------ 1 root root 64 Jun 26 04:26 17 -> /dev/urandom
lr-x------ 1 root root 64 Jun 26 04:26 18 -> /dev/random
lr-x------ 1 root root 64 Jun 26 04:26 19 -> /dev/random
l-wx------ 1 root root 64 Jun 26 04:26 2 -> /root/docker/HelloWorld/logs/nohup.log
lr-x------ 1 root root 64 Jun 26 04:26 20 -> /dev/urandom
lr-x------ 1 root root 64 Jun 26 04:26 21 -> /dev/urandom
lr-x------ 1 root root 64 Jun 26 04:26 22 -> /usr/local/ProgramFiles/jdk1.8.0_291/jre/lib/ext/cldrdata.jar
lr-x------ 1 root root 64 Jun 26 04:26 23 -> /usr/local/ProgramFiles/jdk1.8.0_291/jre/lib/ext/localedata.jar
lrwx------ 1 root root 64 Jun 26 04:26 24 -> socket:[6226176]
lr-x------ 1 root root 64 Jun 26 04:26 25 -> pipe:[6230147]
lr-x------ 1 root root 64 Jun 26 04:26 26 -> /usr/local/ProgramFiles/jdk1.8.0_291/jre/lib/jce.jar
lr-x------ 1 root root 64 Jun 26 04:26 27 -> /usr/local/ProgramFiles/jdk1.8.0_291/jre/lib/ext/sunpkcs11.jar
lr-x------ 1 root root 64 Jun 26 04:26 28 -> /usr/local/ProgramFiles/jdk1.8.0_291/jre/lib/ext/sunec.jar
lr-x------ 1 root root 64 Jun 26 04:26 29 -> /usr/local/ProgramFiles/jdk1.8.0_291/jre/lib/ext/sunjce_provider.jar
lr-x------ 1 root root 64 Jun 26 04:26 3 -> /usr/local/ProgramFiles/jdk1.8.0_291/jre/lib/rt.jar
lrwx------ 1 root root 64 Jun 26 04:26 30 -> socket:[6223845]
lr-x------ 1 root root 64 Jun 26 04:26 31 -> /usr/local/ProgramFiles/jdk1.8.0_291/jre/lib/resources.jar
lrwx------ 1 root root 64 Jun 26 04:26 32 -> socket:[6223868]
lrwx------ 1 root root 64 Jun 26 04:26 33 -> socket:[6223871]
lr-x------ 1 root root 64 Jun 26 04:26 34 -> /usr/local/ProgramFiles/jdk1.8.0_291/jre/lib/charsets.jar
l-wx------ 1 root root 64 Jun 26 04:36 35 -> pipe:[6230147]
lrwx------ 1 root root 64 Jun 26 04:36 36 -> anon_inode:[eventpoll]
lr-x------ 1 root root 64 Jun 26 04:40 37 -> pipe:[6230350]
l-wx------ 1 root root 64 Jun 26 04:40 38 -> pipe:[6230350]
lrwx------ 1 root root 64 Jun 26 04:40 39 -> anon_inode:[eventpoll]
l-wx------ 1 root root 64 Jun 26 04:26 4 -> /root/docker/HelloWorld/logs/debug-2023-06-26.0.log
lrwx------ 1 root root 64 Jun 26 04:26 5 -> socket:[6223797]
lr-x------ 1 root root 64 Jun 26 04:26 6 -> /root/docker/HelloWorld/HelloWorld.jar
lr-x------ 1 root root 64 Jun 26 04:26 7 -> /root/docker/HelloWorld/HelloWorld.jar
l-wx------ 1 root root 64 Jun 26 04:26 8 -> /root/docker/HelloWorld/logs/info-2023-06-26.0.log
l-wx------ 1 root root 64 Jun 26 04:26 9 -> /root/docker/HelloWorld/logs/warn-2023-06-26.0.log
数量大致 规则是这样
当前 ClusterConnectionManager 的 super(config) 会创建 8[未配置线程数量, 使用的默认值] 个 NioEventLoop
下面遍历配置的 redis 集群列表, 每一个服务创建一组 NioEventLoop
遍历 redis 集群配置列表, 依次连接, 在 RedisClientConfig 中有创建 NioEventLoop, 这里是 创建的 8 个[构造方法中调用的 NioEventLoopGroup 的无参构造方法, 在 MultithreadEventLoopGroup 中取的默认值]
RedisClientConfig 中初始化 NioEventLoopGroup 的地方
p ipe 和 anon_inode 创建的地方
EPollSelectorProvider 初始化的地方
EPollSelectorImpl 创建了一对 pipe 的 fd, EPollArrayWrapper 中创建的 anon_inode 的 fd
EPollArrayWrapper 中创建的 epoll 对应的 fd
更基础的东西就是 pipe, epoll_create 等等操作系统的系统调用了, 这里不多赘述
完