SpringCloudGateway 3.1.4版本 Netty内存泄漏问题解决

一、 产生的异常

当时是服务器访问不到服务了,上去一看,无法申请资源OutOfDirectMemoryError了,内存级别的东西让人一阵头大,赶紧在线下模拟,

  1. 减少分配的堆外内存,打开Netty的监测工具等有助于复现的配置和检查的信息

-XX:MaxDirectMemorySize=2M

-Dio.netty.leakDetection.acquireAndReleaseOnly=true

-Dio.netty.leakDetectionLevel=paranoid

-Dio.netty.leakDetection.targetRecords=50

-XX:-OmitStackTraceInFastThrow

-XX:NativeMemoryTracking=detail

Netty的内存泄漏检测分为四个级别:

  • DISABLED - 完成禁止检测内存泄漏,这个是不推荐。
  • SIMPLE - 如果buffer中出现1%的内存泄漏,打印错误日志,就是上面那样。但是日志是看不到详细在什么位置出现了内存泄漏。鉴于性能考虑,这个是在生产环境中默认使用。
  • ADVANCED - 如果buffer的1%出现内存泄漏,打印错误,并且输出详细的内存泄漏信息。
  • PARANOID - 这个和ADVANCED输出的内容是一样的,但它会对每一次请求的buffer做检查。很适用于调试和单元测试。

建议

1、为了今早的发现问题,强烈建议在开发调试过程中,把检测级别设为PARANOID,设置:

io.netty.leakDetection.level=PARANOID
java 复制代码
2023-12-19 17:33:47.568 ERROR 19740 --- [tor-http-nio-54] reactor.netty.transport.ServerTransport  : [158851ed, L:/127.0.0.1:10101 - R:/127.0.0.1:58747] onUncaughtException(SimpleConnection{channel=[id: 0x158851ed, L:/127.0.0.1:10101 - R:/127.0.0.1:58747]})

io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 2048 byte(s) of direct memory (used: 2095623, max: 2097152)
	at io.netty.util.internal.PlatformDependent.incrementMemoryCounter(PlatformDependent.java:845) ~[netty-common-4.1.84.Final.jar:4.1.84.Final]
	at io.netty.util.internal.PlatformDependent.allocateDirectNoCleaner(PlatformDependent.java:774) ~[netty-common-4.1.84.Final.jar:4.1.84.Final]
	at io.netty.buffer.UnpooledUnsafeNoCleanerDirectByteBuf.allocateDirect(UnpooledUnsafeNoCleanerDirectByteBuf.java:30) ~[netty-buffer-4.1.84.Final.jar:4.1.84.Final]
	at io.netty.buffer.UnpooledDirectByteBuf.<init>(UnpooledDirectByteBuf.java:64) ~[netty-buffer-4.1.84.Final.jar:4.1.84.Final]
	at io.netty.buffer.UnpooledUnsafeDirectByteBuf.<init>(UnpooledUnsafeDirectByteBuf.java:41) ~[netty-buffer-4.1.84.Final.jar:4.1.84.Final]
	at io.netty.buffer.UnpooledUnsafeNoCleanerDirectByteBuf.<init>(UnpooledUnsafeNoCleanerDirectByteBuf.java:25) ~[netty-buffer-4.1.84.Final.jar:4.1.84.Final]
	at io.netty.buffer.UnsafeByteBufUtil.newUnsafeDirectByteBuf(UnsafeByteBufUtil.java:632) ~[netty-buffer-4.1.84.Final.jar:4.1.84.Final]
	at io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:399) ~[netty-buffer-4.1.84.Final.jar:4.1.84.Final]
	at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:188) ~[netty-buffer-4.1.84.Final.jar:4.1.84.Final]
	at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:179) ~[netty-buffer-4.1.84.Final.jar:4.1.84.Final]
	at io.netty.buffer.AbstractByteBufAllocator.ioBuffer(AbstractByteBufAllocator.java:140) ~[netty-buffer-4.1.84.Final.jar:4.1.84.Final]
	at io.netty.channel.DefaultMaxMessagesRecvByteBufAllocator$MaxMessageHandle.allocate(DefaultMaxMessagesRecvByteBufAllocator.java:120) ~[netty-transport-4.1.84.Final.jar:4.1.84.Final]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:150) ~[netty-transport-4.1.84.Final.jar:4.1.84.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) [netty-transport-4.1.84.Final.jar:4.1.84.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) [netty-transport-4.1.84.Final.jar:4.1.84.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) [netty-transport-4.1.84.Final.jar:4.1.84.Final]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) [netty-transport-4.1.84.Final.jar:4.1.84.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) [netty-common-4.1.84.Final.jar:4.1.84.Final]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.84.Final.jar:4.1.84.Final]
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-common-4.1.84.Final.jar:4.1.84.Final]
	at java.lang.Thread.run(Thread.java:750) [na:1.8.0_351]

二、 检查内存泄漏点位

由于有点多,还特意把检测的详细条数改到了50。

挨个查看是不是有自己写的代码或者熟悉的代码,看了一遍没认识的,完蛋这会麻烦了,各种找资料(百度),上官网(Netty的Issues · netty/netty · GitHub)(gateway的LEAK: ByteBuf.release() · Issue #26937 · spring-projects/spring-framework · GitHub)各种查,大部分都是版本问题但是都是老版本,我看我的都是新版的了咋还有,DEBUG后发现总是在请求进来时申请一次2048的,在后续的处理中涨涨降降最后就是这点没释放

java 复制代码
2023-12-19 09:33:12.933 [reactor-http-nio-35] ERROR io.netty.util.ResourceLeakDetector - LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
Recent access records: 
#1:
	io.netty.util.ReferenceCountUtil.release(ReferenceCountUtil.java:90)
	io.netty.util.ReferenceCountUtil.safeRelease(ReferenceCountUtil.java:116)
	io.netty.channel.ChannelOutboundBuffer.remove(ChannelOutboundBuffer.java:271)
	io.netty.channel.ChannelOutboundBuffer.removeBytes(ChannelOutboundBuffer.java:352)
	io.netty.channel.socket.nio.NioSocketChannel.doWrite(NioSocketChannel.java:421)
	io.netty.channel.AbstractChannel$AbstractUnsafe.flush0(AbstractChannel.java:931)
	io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.flush0(AbstractNioChannel.java:354)
	io.netty.channel.AbstractChannel$AbstractUnsafe.flush(AbstractChannel.java:895)
	io.netty.channel.DefaultChannelPipeline$HeadContext.flush(DefaultChannelPipeline.java:1372)
	io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:921)
	io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:907)
	io.netty.channel.AbstractChannelHandlerContext.flush(AbstractChannelHandlerContext.java:893)
	io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.flush(CombinedChannelDuplexHandler.java:531)
	io.netty.channel.ChannelOutboundHandlerAdapter.flush(ChannelOutboundHandlerAdapter.java:125)
	io.netty.channel.CombinedChannelDuplexHandler.flush(CombinedChannelDuplexHandler.java:356)
	io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:923)
	io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:907)
	io.netty.channel.AbstractChannelHandlerContext.flush(AbstractChannelHandlerContext.java:893)
	reactor.netty.channel.MonoSendMany$SendManyInner$AsyncFlush.run(MonoSendMany.java:789)
	io.netty.util.concurrent.AbstractEventExecutor.runTask$$$capture(AbstractEventExecutor.java:174)
	io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java)
	io.netty.util.concurrent.AbstractEventExecutor.safeExecute$$$capture(AbstractEventExecutor.java:167)
	io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java)
	io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470)
	io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:569)
	io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
	io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	java.lang.Thread.run(Thread.java:750)
#2:
	Hint: 'DefaultChannelPipeline$HeadContext#0' will handle the message from this point.
	io.netty.channel.DefaultChannelPipeline.touch(DefaultChannelPipeline.java:116)
	io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:962)
	io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:856)
	io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.write(CombinedChannelDuplexHandler.java:526)
	io.netty.handler.codec.http.HttpObjectEncoder.writeOutList(HttpObjectEncoder.java:118)
	io.netty.handler.codec.http.HttpObjectEncoder.write(HttpObjectEncoder.java:110)
	io.netty.channel.CombinedChannelDuplexHandler.write(CombinedChannelDuplexHandler.java:346)
	io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:879)
	io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:863)
	io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:968)
	io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:856)
	reactor.netty.channel.MonoSendMany$SendManyInner.run(MonoSendMany.java:321)
	reactor.netty.channel.MonoSendMany$SendManyInner.trySchedule(MonoSendMany.java:423)
	reactor.netty.channel.MonoSendMany$SendManyInner.onNext(MonoSendMany.java:223)
	reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:129)
	reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1816)
	reactor.core.publisher.FluxCallable.subscribe(FluxCallable.java:49)
	reactor.core.publisher.Flux.subscribe(Flux.java:8526)
	reactor.netty.channel.MonoSendMany.subscribe(MonoSendMany.java:102)
	reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.subscribeNext(MonoIgnoreThen.java:240)
	reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.onComplete(MonoIgnoreThen.java:203)
	reactor.core.publisher.Operators.complete(Operators.java:137)
	reactor.netty.FutureMono.doSubscribe(FutureMono.java:122)
	reactor.netty.FutureMono$DeferredFutureMono.subscribe(FutureMono.java:114)
	reactor.core.publisher.Mono.subscribe(Mono.java:4455)
	reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.subscribeNext(MonoIgnoreThen.java:263)
	reactor.core.publisher.MonoIgnoreThen.subscribe(MonoIgnoreThen.java:51)
	reactor.core.publisher.Mono.subscribe(Mono.java:4455)
	reactor.netty.NettyOutbound.subscribe(NettyOutbound.java:336)
	reactor.core.publisher.MonoSource.subscribe(MonoSource.java:69)
	reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52)
	reactor.netty.http.client.HttpClientConnect$HttpIOHandlerObserver.onStateChange(HttpClientConnect.java:441)
	reactor.netty.ReactorNetty$CompositeConnectionObserver.onStateChange(ReactorNetty.java:677)
	reactor.netty.resources.DefaultPooledConnectionProvider$DisposableAcquire.run(DefaultPooledConnectionProvider.java:274)
	io.netty.util.concurrent.AbstractEventExecutor.runTask$$$capture(AbstractEventExecutor.java:174)
	io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java)
	io.netty.util.concurrent.AbstractEventExecutor.safeExecute$$$capture(AbstractEventExecutor.java:167)
	io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java)
	io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470)
	io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:569)
	io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
	io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	java.lang.Thread.run(Thread.java:750)
#3:
	io.netty.handler.codec.http.HttpObjectEncoder.encodeByteBufContent(HttpObjectEncoder.java:405)
	io.netty.handler.codec.http.HttpObjectEncoder.encodeNotHttpMessageContentTypes(HttpObjectEncoder.java:285)
	io.netty.handler.codec.http.HttpObjectEncoder.encode(HttpObjectEncoder.java:181)
	io.netty.handler.codec.http.HttpClientCodec$Encoder.encode(HttpClientCodec.java:203)
	io.netty.handler.codec.http.HttpObjectEncoder.write(HttpObjectEncoder.java:97)
	io.netty.channel.CombinedChannelDuplexHandler.write(CombinedChannelDuplexHandler.java:346)
	io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:879)
	io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:863)
	io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:968)
	io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:856)
	reactor.netty.channel.MonoSendMany$SendManyInner.run(MonoSendMany.java:321)
	reactor.netty.channel.MonoSendMany$SendManyInner.trySchedule(MonoSendMany.java:423)
	reactor.netty.channel.MonoSendMany$SendManyInner.onNext(MonoSendMany.java:223)
	reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:129)
	reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1816)
	reactor.core.publisher.FluxCallable.subscribe(FluxCallable.java:49)
	reactor.core.publisher.Flux.subscribe(Flux.java:8526)
	reactor.netty.channel.MonoSendMany.subscribe(MonoSendMany.java:102)
	reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.subscribeNext(MonoIgnoreThen.java:240)
	reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.onComplete(MonoIgnoreThen.java:203)
	reactor.core.publisher.Operators.complete(Operators.java:137)
	reactor.netty.FutureMono.doSubscribe(FutureMono.java:122)
	reactor.netty.FutureMono$DeferredFutureMono.subscribe(FutureMono.java:114)
	reactor.core.publisher.Mono.subscribe(Mono.java:4455)
	reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.subscribeNext(MonoIgnoreThen.java:263)
	reactor.core.publisher.MonoIgnoreThen.subscribe(MonoIgnoreThen.java:51)
	reactor.core.publisher.Mono.subscribe(Mono.java:4455)
	reactor.netty.NettyOutbound.subscribe(NettyOutbound.java:336)
	reactor.core.publisher.MonoSource.subscribe(MonoSource.java:69)
	reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52)
	reactor.netty.http.client.HttpClientConnect$HttpIOHandlerObserver.onStateChange(HttpClientConnect.java:441)
	reactor.netty.ReactorNetty$CompositeConnectionObserver.onStateChange(ReactorNetty.java:677)
	reactor.netty.resources.DefaultPooledConnectionProvider$DisposableAcquire.run(DefaultPooledConnectionProvider.java:274)
	io.netty.util.concurrent.AbstractEventExecutor.runTask$$$capture(AbstractEventExecutor.java:174)
	io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java)
	io.netty.util.concurrent.AbstractEventExecutor.safeExecute$$$capture(AbstractEventExecutor.java:167)
	io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java)
	io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470)
	io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:569)
	io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
	io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	java.lang.Thread.run(Thread.java:750)
#4:
	io.netty.handler.codec.http.HttpObjectEncoder.encodeByteBufAndTrailers(HttpObjectEncoder.java:453)
	io.netty.handler.codec.http.HttpObjectEncoder.encodeByteBufContent(HttpObjectEncoder.java:403)
	io.netty.handler.codec.http.HttpObjectEncoder.encodeNotHttpMessageContentTypes(HttpObjectEncoder.java:285)
	io.netty.handler.codec.http.HttpObjectEncoder.encode(HttpObjectEncoder.java:181)
	io.netty.handler.codec.http.HttpClientCodec$Encoder.encode(HttpClientCodec.java:203)
	io.netty.handler.codec.http.HttpObjectEncoder.write(HttpObjectEncoder.java:97)
	io.netty.channel.CombinedChannelDuplexHandler.write(CombinedChannelDuplexHandler.java:346)
	io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:879)
	io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:863)
	io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:968)
	io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:856)
	reactor.netty.channel.MonoSendMany$SendManyInner.run(MonoSendMany.java:321)
	reactor.netty.channel.MonoSendMany$SendManyInner.trySchedule(MonoSendMany.java:423)
	reactor.netty.channel.MonoSendMany$SendManyInner.onNext(MonoSendMany.java:223)
	reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:129)
	reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1816)
	reactor.core.publisher.FluxCallable.subscribe(FluxCallable.java:49)
	reactor.core.publisher.Flux.subscribe(Flux.java:8526)
	reactor.netty.channel.MonoSendMany.subscribe(MonoSendMany.java:102)
	reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.subscribeNext(MonoIgnoreThen.java:240)
	reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.onComplete(MonoIgnoreThen.java:203)
	reactor.core.publisher.Operators.complete(Operators.java:137)
	reactor.netty.FutureMono.doSubscribe(FutureMono.java:122)
	reactor.netty.FutureMono$DeferredFutureMono.subscribe(FutureMono.java:114)
	reactor.core.publisher.Mono.subscribe(Mono.java:4455)
	reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.subscribeNext(MonoIgnoreThen.java:263)
	reactor.core.publisher.MonoIgnoreThen.subscribe(MonoIgnoreThen.java:51)
	reactor.core.publisher.Mono.subscribe(Mono.java:4455)
	reactor.netty.NettyOutbound.subscribe(NettyOutbound.java:336)
	reactor.core.publisher.MonoSource.subscribe(MonoSource.java:69)
	reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52)
	reactor.netty.http.client.HttpClientConnect$HttpIOHandlerObserver.onStateChange(HttpClientConnect.java:441)
	reactor.netty.ReactorNetty$CompositeConnectionObserver.onStateChange(ReactorNetty.java:677)
	reactor.netty.resources.DefaultPooledConnectionProvider$DisposableAcquire.run(DefaultPooledConnectionProvider.java:274)
	io.netty.util.concurrent.AbstractEventExecutor.runTask$$$capture(AbstractEventExecutor.java:174)
	io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java)
	io.netty.util.concurrent.AbstractEventExecutor.safeExecute$$$capture(AbstractEventExecutor.java:167)
	io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java)
	io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470)
	io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:569)
	io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
	io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	java.lang.Thread.run(Thread.java:750)
#5:
	Hint: 'reactor.left.httpCodec' will handle the message from this point.
	io.netty.channel.DefaultChannelPipeline.touch(DefaultChannelPipeline.java:116)
	io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:962)
	io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:856)
	reactor.netty.channel.MonoSendMany$SendManyInner.run(MonoSendMany.java:321)
	reactor.netty.channel.MonoSendMany$SendManyInner.trySchedule(MonoSendMany.java:423)
	reactor.netty.channel.MonoSendMany$SendManyInner.onNext(MonoSendMany.java:223)
	reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:129)
	reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1816)
	reactor.core.publisher.FluxCallable.subscribe(FluxCallable.java:49)
	reactor.core.publisher.Flux.subscribe(Flux.java:8526)
	reactor.netty.channel.MonoSendMany.subscribe(MonoSendMany.java:102)
	reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.subscribeNext(MonoIgnoreThen.java:240)
	reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.onComplete(MonoIgnoreThen.java:203)
	reactor.core.publisher.Operators.complete(Operators.java:137)
	reactor.netty.FutureMono.doSubscribe(FutureMono.java:122)
	reactor.netty.FutureMono$DeferredFutureMono.subscribe(FutureMono.java:114)
	reactor.core.publisher.Mono.subscribe(Mono.java:4455)
	reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.subscribeNext(MonoIgnoreThen.java:263)
	reactor.core.publisher.MonoIgnoreThen.subscribe(MonoIgnoreThen.java:51)
	reactor.core.publisher.Mono.subscribe(Mono.java:4455)
	reactor.netty.NettyOutbound.subscribe(NettyOutbound.java:336)
	reactor.core.publisher.MonoSource.subscribe(MonoSource.java:69)
	reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52)
	reactor.netty.http.client.HttpClientConnect$HttpIOHandlerObserver.onStateChange(HttpClientConnect.java:441)
	reactor.netty.ReactorNetty$CompositeConnectionObserver.onStateChange(ReactorNetty.java:677)
	reactor.netty.resources.DefaultPooledConnectionProvider$DisposableAcquire.run(DefaultPooledConnectionProvider.java:274)
	io.netty.util.concurrent.AbstractEventExecutor.runTask$$$capture(AbstractEventExecutor.java:174)
	io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java)
	io.netty.util.concurrent.AbstractEventExecutor.safeExecute$$$capture(AbstractEventExecutor.java:167)
	io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java)
	io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470)
	io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:569)
	io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
	io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	java.lang.Thread.run(Thread.java:750)
#6:
	org.springframework.core.io.buffer.NettyDataBufferFactory.wrap(NettyDataBufferFactory.java:93)
	org.springframework.cloud.gateway.support.ServerWebExchangeUtils$1.lambda$getBody$0(ServerWebExchangeUtils.java:375)
	reactor.core.publisher.MonoSupplier.call(MonoSupplier.java:86)
	reactor.core.publisher.FluxCallable.subscribe(FluxCallable.java:44)
	reactor.core.publisher.Flux.subscribe(Flux.java:8526)
	reactor.netty.channel.MonoSendMany.subscribe(MonoSendMany.java:102)
	reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.subscribeNext(MonoIgnoreThen.java:240)
	reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.onComplete(MonoIgnoreThen.java:203)
	reactor.core.publisher.Operators.complete(Operators.java:137)
	reactor.netty.FutureMono.doSubscribe(FutureMono.java:122)
	reactor.netty.FutureMono$DeferredFutureMono.subscribe(FutureMono.java:114)
	reactor.core.publisher.Mono.subscribe(Mono.java:4455)
	reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.subscribeNext(MonoIgnoreThen.java:263)
	reactor.core.publisher.MonoIgnoreThen.subscribe(MonoIgnoreThen.java:51)
	reactor.core.publisher.Mono.subscribe(Mono.java:4455)
	reactor.netty.NettyOutbound.subscribe(NettyOutbound.java:336)
	reactor.core.publisher.MonoSource.subscribe(MonoSource.java:69)
	reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52)
	reactor.netty.http.client.HttpClientConnect$HttpIOHandlerObserver.onStateChange(HttpClientConnect.java:441)
	reactor.netty.ReactorNetty$CompositeConnectionObserver.onStateChange(ReactorNetty.java:677)
	reactor.netty.resources.DefaultPooledConnectionProvider$DisposableAcquire.run(DefaultPooledConnectionProvider.java:274)
	io.netty.util.concurrent.AbstractEventExecutor.runTask$$$capture(AbstractEventExecutor.java:174)
	io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java)
	io.netty.util.concurrent.AbstractEventExecutor.safeExecute$$$capture(AbstractEventExecutor.java:167)
	io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java)
	io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470)
	io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:569)
	io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
	io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	java.lang.Thread.run(Thread.java:750)
#7:
	org.springframework.core.io.buffer.NettyDataBuffer.release(NettyDataBuffer.java:326)
	org.springframework.core.io.buffer.DataBufferUtils.release(DataBufferUtils.java:521)
	org.springframework.core.codec.StringDecoder.decode(StringDecoder.java:192)
	org.springframework.core.codec.StringDecoder.decode(StringDecoder.java:61)
	org.springframework.core.codec.AbstractDataBufferDecoder.decodeDataBuffer(AbstractDataBufferDecoder.java:109)
	org.springframework.core.codec.AbstractDataBufferDecoder.lambda$decodeToMono$1(AbstractDataBufferDecoder.java:96)
	reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:113)
	reactor.core.publisher.FluxContextWrite$ContextWriteSubscriber.onNext(FluxContextWrite.java:107)
	reactor.core.publisher.FluxMapFuseable$MapFuseableConditionalSubscriber.onNext(FluxMapFuseable.java:299)
	reactor.core.publisher.FluxFilterFuseable$FilterFuseableConditionalSubscriber.onNext(FluxFilterFuseable.java:337)
	reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1816)
	reactor.core.publisher.MonoCollect$CollectSubscriber.onComplete(MonoCollect.java:160)
	reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1817)
	reactor.core.publisher.FluxCallable.subscribe(FluxCallable.java:49)
	reactor.core.publisher.Mono.subscribe(Mono.java:4455)
	reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.subscribeNext(MonoIgnoreThen.java:263)
	reactor.core.publisher.MonoIgnoreThen.subscribe(MonoIgnoreThen.java:51)
	reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:157)
	reactor.core.publisher.FluxSwitchIfEmpty$SwitchIfEmptySubscriber.onNext(FluxSwitchIfEmpty.java:74)
	reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:122)
	reactor.core.publisher.FluxDefaultIfEmpty$DefaultIfEmptySubscriber.onNext(FluxDefaultIfEmpty.java:101)
	reactor.core.publisher.FluxContextWrite$ContextWriteSubscriber.onNext(FluxContextWrite.java:107)
	reactor.core.publisher.FluxMapFuseable$MapFuseableConditionalSubscriber.onNext(FluxMapFuseable.java:299)
	reactor.core.publisher.FluxFilterFuseable$FilterFuseableConditionalSubscriber.onNext(FluxFilterFuseable.java:337)
	reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1816)
	reactor.core.publisher.MonoCollect$CollectSubscriber.onComplete(MonoCollect.java:160)
	reactor.core.publisher.FluxMap$MapSubscriber.onComplete(FluxMap.java:144)
	reactor.core.publisher.FluxPeek$PeekSubscriber.onComplete(FluxPeek.java:260)
	reactor.core.publisher.FluxMap$MapSubscriber.onComplete(FluxMap.java:144)
	reactor.netty.channel.FluxReceive.onInboundComplete(FluxReceive.java:400)
	reactor.netty.channel.ChannelOperations.onInboundComplete(ChannelOperations.java:419)
	reactor.netty.http.server.HttpServerOperations.onInboundNext(HttpServerOperations.java:603)
	reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:113)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	reactor.netty.http.server.HttpTrafficHandler.channelRead(HttpTrafficHandler.java:266)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436)
	io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:336)
	io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:308)
	io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
	io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
	io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
	io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
	io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
	io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
	io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
	io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	java.lang.Thread.run(Thread.java:750)
#8:
	org.springframework.core.io.buffer.NettyDataBufferFactory.wrap(NettyDataBufferFactory.java:93)
	org.springframework.cloud.gateway.support.ServerWebExchangeUtils$1.lambda$getBody$0(ServerWebExchangeUtils.java:375)
	reactor.core.publisher.MonoSupplier.call(MonoSupplier.java:86)
	reactor.core.publisher.FluxCallable.subscribe(FluxCallable.java:44)
	reactor.core.publisher.Mono.subscribe(Mono.java:4455)
	reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.subscribeNext(MonoIgnoreThen.java:263)
	reactor.core.publisher.MonoIgnoreThen.subscribe(MonoIgnoreThen.java:51)
	reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:157)
	reactor.core.publisher.FluxSwitchIfEmpty$SwitchIfEmptySubscriber.onNext(FluxSwitchIfEmpty.java:74)
	reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:122)
	reactor.core.publisher.FluxDefaultIfEmpty$DefaultIfEmptySubscriber.onNext(FluxDefaultIfEmpty.java:101)
	reactor.core.publisher.FluxContextWrite$ContextWriteSubscriber.onNext(FluxContextWrite.java:107)
	reactor.core.publisher.FluxMapFuseable$MapFuseableConditionalSubscriber.onNext(FluxMapFuseable.java:299)
	reactor.core.publisher.FluxFilterFuseable$FilterFuseableConditionalSubscriber.onNext(FluxFilterFuseable.java:337)
	reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1816)
	reactor.core.publisher.MonoCollect$CollectSubscriber.onComplete(MonoCollect.java:160)
	reactor.core.publisher.FluxMap$MapSubscriber.onComplete(FluxMap.java:144)
	reactor.core.publisher.FluxPeek$PeekSubscriber.onComplete(FluxPeek.java:260)
	reactor.core.publisher.FluxMap$MapSubscriber.onComplete(FluxMap.java:144)
	reactor.netty.channel.FluxReceive.onInboundComplete(FluxReceive.java:400)
	reactor.netty.channel.ChannelOperations.onInboundComplete(ChannelOperations.java:419)
	reactor.netty.http.server.HttpServerOperations.onInboundNext(HttpServerOperations.java:603)
	reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:113)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	reactor.netty.http.server.HttpTrafficHandler.channelRead(HttpTrafficHandler.java:266)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436)
	io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:336)
	io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:308)
	io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
	io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
	io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
	io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
	io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
	io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
	io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
	io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	java.lang.Thread.run(Thread.java:750)
#9:
	io.netty.handler.codec.http.DefaultHttpContent.release(DefaultHttpContent.java:92)
	io.netty.util.ReferenceCountUtil.release(ReferenceCountUtil.java:90)
	reactor.netty.channel.FluxReceive.onInboundNext(FluxReceive.java:367)
	reactor.netty.channel.ChannelOperations.onInboundNext(ChannelOperations.java:404)
	reactor.netty.http.server.HttpServerOperations.onInboundNext(HttpServerOperations.java:598)
	reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:113)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	reactor.netty.http.server.HttpTrafficHandler.channelRead(HttpTrafficHandler.java:266)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436)
	io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:336)
	io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:308)
	io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
	io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
	io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
	io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
	io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
	io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
	io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
	io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	java.lang.Thread.run(Thread.java:750)
#10:
	org.springframework.core.io.buffer.NettyDataBufferFactory.wrap(NettyDataBufferFactory.java:93)
	reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:106)
	reactor.core.publisher.FluxPeek$PeekSubscriber.onNext(FluxPeek.java:200)
	reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:122)
	reactor.netty.channel.FluxReceive.onInboundNext(FluxReceive.java:364)
	reactor.netty.channel.ChannelOperations.onInboundNext(ChannelOperations.java:404)
	reactor.netty.http.server.HttpServerOperations.onInboundNext(HttpServerOperations.java:598)
	reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:113)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	reactor.netty.http.server.HttpTrafficHandler.channelRead(HttpTrafficHandler.java:266)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436)
	io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:336)
	io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:308)
	io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
	io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
	io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
	io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
	io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
	io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
	io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
	io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	java.lang.Thread.run(Thread.java:750)
#11:
	reactor.core.publisher.FluxPeek$PeekSubscriber.onNext(FluxPeek.java:185)
	reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:122)
	reactor.netty.channel.FluxReceive.onInboundNext(FluxReceive.java:364)
	reactor.netty.channel.ChannelOperations.onInboundNext(ChannelOperations.java:404)
	reactor.netty.http.server.HttpServerOperations.onInboundNext(HttpServerOperations.java:598)
	reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:113)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	reactor.netty.http.server.HttpTrafficHandler.channelRead(HttpTrafficHandler.java:266)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436)
	io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:336)
	io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:308)
	io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
	io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
	io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
	io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
	io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
	io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
	io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
	io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	java.lang.Thread.run(Thread.java:750)
#12:
	Hint: [a2330415-3, L:/127.0.0.1:10101 - R:/127.0.0.1:52736] Receiver reactor.core.publisher.FluxMap$MapSubscriber will handle the message from this point
	io.netty.handler.codec.http.DefaultHttpContent.touch(DefaultHttpContent.java:86)
	io.netty.handler.codec.http.DefaultLastHttpContent.touch(DefaultLastHttpContent.java:96)
	io.netty.handler.codec.http.DefaultLastHttpContent.touch(DefaultLastHttpContent.java:30)
	reactor.netty.channel.FluxReceive.onInboundNext(FluxReceive.java:360)
	reactor.netty.channel.ChannelOperations.onInboundNext(ChannelOperations.java:404)
	reactor.netty.http.server.HttpServerOperations.onInboundNext(HttpServerOperations.java:598)
	reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:113)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	reactor.netty.http.server.HttpTrafficHandler.channelRead(HttpTrafficHandler.java:266)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436)
	io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:336)
	io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:308)
	io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
	io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
	io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
	io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
	io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
	io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
	io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
	io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	java.lang.Thread.run(Thread.java:750)
#13:
	Hint: 'reactor.right.reactiveBridge' will handle the message from this point.
	io.netty.handler.codec.http.DefaultHttpContent.touch(DefaultHttpContent.java:86)
	io.netty.handler.codec.http.DefaultLastHttpContent.touch(DefaultLastHttpContent.java:96)
	io.netty.handler.codec.http.DefaultLastHttpContent.touch(DefaultLastHttpContent.java:30)
	io.netty.channel.DefaultChannelPipeline.touch(DefaultChannelPipeline.java:116)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:417)
	io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	reactor.netty.http.server.HttpTrafficHandler.channelRead(HttpTrafficHandler.java:266)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436)
	io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:336)
	io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:308)
	io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
	io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
	io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
	io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
	io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
	io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
	io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
	io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	java.lang.Thread.run(Thread.java:750)
#14:
	Hint: 'reactor.left.httpTrafficHandler' will handle the message from this point.
	io.netty.handler.codec.http.DefaultHttpContent.touch(DefaultHttpContent.java:86)
	io.netty.handler.codec.http.DefaultLastHttpContent.touch(DefaultLastHttpContent.java:96)
	io.netty.handler.codec.http.DefaultLastHttpContent.touch(DefaultLastHttpContent.java:30)
	io.netty.channel.DefaultChannelPipeline.touch(DefaultChannelPipeline.java:116)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:417)
	io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436)
	io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:336)
	io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:308)
	io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
	io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
	io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
	io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
	io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
	io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
	io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
	io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	java.lang.Thread.run(Thread.java:750)
#15:
	io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290)
	io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
	io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
	io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
	io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
	io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
	io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
	io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
	io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	java.lang.Thread.run(Thread.java:750)
#16:
	Hint: 'reactor.left.httpCodec' will handle the message from this point.
	io.netty.channel.DefaultChannelPipeline.touch(DefaultChannelPipeline.java:116)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:417)
	io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
	io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
	io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
	io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
	io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
	io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
	io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
	io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	java.lang.Thread.run(Thread.java:750)
#17:
	Hint: 'DefaultChannelPipeline$HeadContext#0' will handle the message from this point.
	io.netty.channel.DefaultChannelPipeline.touch(DefaultChannelPipeline.java:116)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:417)
	io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
	io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
	io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
	io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
	io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
	io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
	io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
	io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	java.lang.Thread.run(Thread.java:750)
Created at:
	io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:403)
	io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:188)
	io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:179)
	io.netty.buffer.AbstractByteBufAllocator.ioBuffer(AbstractByteBufAllocator.java:140)
	io.netty.channel.DefaultMaxMessagesRecvByteBufAllocator$MaxMessageHandle.allocate(DefaultMaxMessagesRecvByteBufAllocator.java:120)
	io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:150)
	io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
	io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
	io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
	io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
	io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
	io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	java.lang.Thread.run(Thread.java:750)

三、 修复

别看上面一堆很长,关键的就是几个点:

Hint: 开头的

Spring的一些

如果有自己的代码记得释放就好,我的没有就需要对比试用版本和泄漏点是不是有使用没有释放,或者释放未成功。

第一次试着升级了gateway的版本但是无效,后来想单独升级会不会有冲突,回滚后一起升级的Springboot+SpringCloud,试运行后还是不行,查看依赖时发现Libraries中旧版Jar包引入在未删除,造成了冲突,删除后,Maven刷新,在运行一切就正常了。

原版本:

复制代码
<spring-boot.version>2.6.13</spring-boot.version>
<spring-cloud-alibaba.version>2021.0.5.0</spring-cloud-alibaba.version>
<spring-cloud.version>2021.0.5</spring-cloud.version>

升级后版本:

复制代码
<spring-boot.version>2.6.15</spring-boot.version>
<spring-cloud-alibaba.version>2021.0.5.0</spring-cloud-alibaba.version>
<spring-cloud.version>2021.0.8</spring-cloud.version>

推荐堆外内存监控程序和检查的一些资料可以参考:

spring gateway LEAK: ByteBuf.release() was not called before it's garbage-collected - 望)=天涯 - 博客园 (cnblogs.com)

SpringCloud-Gateway OutOfDirectMemoryError排查-CSDN博客

Spring Boot各版本与Java版本的对应兼容关系,与构建工具(Maven、Gradle)版本的对应兼容关系,对servlet 容器的支持-CSDN博客

还有一些找不到了,每个里面找个有用的汇总起来解决;希望可以帮到有需要的朋友

相关推荐
小黄编程快乐屋1 小时前
各个排序算法基础速通万字介绍
java·算法·排序算法
材料苦逼不会梦到计算机白富美3 小时前
贪心算法-区间问题 C++
java·c++·贪心算法
小小李程序员7 小时前
LRU缓存
java·spring·缓存
cnsxjean7 小时前
SpringBoot集成Minio实现上传凭证、分片上传、秒传和断点续传
java·前端·spring boot·分布式·后端·中间件·架构
hadage2337 小时前
--- stream 数据流 java ---
java·开发语言
《源码好优多》7 小时前
基于Java Springboot汽配销售管理系统
java·开发语言·spring boot
kingbal8 小时前
SpringCloud:Injection of resource dependencies failed
后端·spring·spring cloud
小林想被监督学习8 小时前
Java后端如何进行文件上传和下载 —— 本地版
java·开发语言
Erosion20208 小时前
SPI机制
java·java sec
逸风尊者9 小时前
开发也能看懂的大模型:RNN
java·后端·算法