dinky提交flink任务报 java.lang.OutOfMemoryError: Direct buffer memory

1，报错信息

org.apache.hadoop.ipc.Client(1093): Unexpected error reading responses on connection ThreadIPC Client (815834176) connection to prod52.bigdata/xx.xx.x.xx:8020 from appuser,5,main java.lang.OutOfMemoryError: Direct buffer memory

at java.nio.Bits.reserveMemory(Bits.java:694) ~?:1.8.0_201 at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123) ~?:1.8.0_201 at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311) ~?:1.8.0_201 at sun.nio.ch.Util.getTemporaryDirectBuffer(Util.java:241) ~?:1.8.0_201 at sun.nio.ch.IOUtil.read(IOUtil.java:195) ~?:1.8.0_201 at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380) ~?:1.8.0_201 at org.apache.hadoop.net.SocketInputStreamReader.performIO(SocketInputStream.java:57) \~\[flink-shaded-hadoop-3-uber-3.1.1.7.2.9.0-173-9.0.jar:3.1.1.7.2.9.0-173-9.0\] at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142) \~\[flink-shaded-hadoop-3-uber-3.1.1.7.2.9.0-173-9.0.jar:3.1.1.7.2.9.0-173-9.0\] at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) \~\[flink-shaded-hadoop-3-uber-3.1.1.7.2.9.0-173-9.0.jar:3.1.1.7.2.9.0-173-9.0\] at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) \~\[flink-shaded-hadoop-3-uber-3.1.1.7.2.9.0-173-9.0.jar:3.1.1.7.2.9.0-173-9.0\] at java.io.FilterInputStream.read(FilterInputStream.java:133) \~\[?:1.8.0_201\] at java.io.BufferedInputStream.read1(BufferedInputStream.java:284) \~\[?:1.8.0_201\] at java.io.BufferedInputStream.read(BufferedInputStream.java:345) \~\[?:1.8.0_201\] at java.io.DataInputStream.read(DataInputStream.java:149) \~\[?:1.8.0_201\] at java.io.FilterInputStream.read(FilterInputStream.java:133) \~\[?:1.8.0_201\] at org.apache.hadoop.ipc.ClientConnectionPingInputStream.read(Client.java:581) \~\[flink-shaded-hadoop-3-uber-3.1.1.7.2.9.0-173-9.0.jar:3.1.1.7.2.9.0-173-9.0\] at java.io.DataInputStream.readFully(DataInputStream.java:195) \~\[?:1.8.0_201\] at java.io.DataInputStream.readFully(DataInputStream.java:169) \~\[?:1.8.0_201\] at org.apache.hadoop.ipc.ClientIpcStreams.readResponse(Client.java:1897) ~flink-shaded-hadoop-3-uber-3.1.1.7.2.9.0-173-9.0.jar:3.1.1.7.2.9.0-173-9.0 at org.apache.hadoop.ipc.ClientConnection.receiveRpcResponse(Client.java:1191) \~\[flink-shaded-hadoop-3-uber-3.1.1.7.2.9.0-173-9.0.jar:3.1.1.7.2.9.0-173-9.0\] at org.apache.hadoop.ipc.ClientConnection.run(Client.java:1087) ~flink-shaded-hadoop-3-uber-3.1.1.7.2.9.0-173-9.0.jar:3.1.1.7.2.9.0-173-9.0

2，报错原因

Dinky 在提交 Flink 任务到 YARN 集群时，因 java.lang.OutOfMemoryError: Direct buffer memory 错误失败。

这个错误发生在 Dinky 服务进程本身（不是 Flink JobManager 或 TaskManager），具体是在它通过 Hadoop RPC 客户端连接 YARN ResourceManager时，尝试分配 Direct Memory（堆外内存）失败**。**

java.lang.OutOfMemoryError: Direct buffer memory

表示 JVM 的 Direct Memory（堆外内存）耗尽 。

Direct Memory 不受 -Xmx 控制，而是由 -XX:MaxDirectMemorySize 限制（默认等于 -Xmx）。

Dinky 在提交任务时：

1. 加载了 flink-shaded-hadoop-3-uber-*.jar
2. 使用 Hadoop 客户端与 YARN RM 通信（IPC）
3. Hadoop RPC 使用 DirectByteBuffer 进行网络 I/O（提升性能）
4. Direct Memory 不足 → 抛出 OOM

3，解决方案

为 Dinky JVM 显式设置 -XX:MaxDirectMemorySize

JAVA_OPTS="-Xmx4g -Xms4g -XX:MaxDirectMemorySize=2g -XX:+UseG1GC"