- 现象是这样的,服务有时启动正常,有时启动失败,最后排查发现是mq消费者造成的。 启动dubbo过程中,mq消费者正常消费,调用某个dubbo服务。此时dubbo application正处于starting状态,造成了死锁。
2.模拟死锁过程,获取到的线程信息,已省略无关的信息。
java
Found one Java-level deadlock:
=============================
"Thread-2":
waiting to lock monitor 0x00000136bfd3bb88 (object 0x00000005c4c594f0, a org.apache.dubbo.config.deploy.DefaultModuleDeployer),
which is held by "main"
"main":
waiting to lock monitor 0x00000136ba6465d8 (object 0x00000005c46287e0, a java.util.concurrent.ConcurrentHashMap),
which is held by "Thread-2"
Java stack information for the threads listed above:
===================================================
"Thread-2":
at org.apache.dubbo.config.deploy.DefaultModuleDeployer.startSync(DefaultModuleDeployer.java:143)
- waiting to lock <0x00000005c4c594f0> (a org.apache.dubbo.config.deploy.DefaultModuleDeployer)
at org.apache.dubbo.config.deploy.DefaultModuleDeployer.start(DefaultModuleDeployer.java:139)
at org.apache.dubbo.config.ReferenceConfig.get(ReferenceConfig.java:228)
at org.apache.dubbo.config.spring.ReferenceBean.getCallProxy(ReferenceBean.java:351)
- locked <0x00000005c46287e0> (a java.util.concurrent.ConcurrentHashMap)
at org.apache.dubbo.config.spring.ReferenceBean.access$100(ReferenceBean.java:100)
at org.apache.dubbo.config.spring.ReferenceBean$DubboReferenceLazyInitTargetSource.createObject(ReferenceBean.java:359)
at org.springframework.aop.target.AbstractLazyCreationTargetSource.getTarget(AbstractLazyCreationTargetSource.java:86)
- locked <0x00000005c98d68d0> (a org.apache.dubbo.config.spring.ReferenceBean$DubboReferenceLazyInitTargetSource)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:192)
....
"main":
at org.springframework.context.event.AbstractApplicationEventMulticaster.getApplicationListeners(AbstractApplicationEventMulticaster.java:187)
- waiting to lock <0x00000005c46287e0> (a java.util.concurrent.ConcurrentHashMap)
at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:128)
at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:393)
at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:347)
at org.apache.dubbo.config.spring.ServiceBean.publishExportEvent(ServiceBean.java:133)
at org.apache.dubbo.config.spring.ServiceBean.exported(ServiceBean.java:125)
at org.apache.dubbo.config.ServiceConfig.doExport(ServiceConfig.java:392)
- locked <0x00000005c9398570> (a org.apache.dubbo.config.spring.ServiceBean)
at org.apache.dubbo.config.ServiceConfig.export(ServiceConfig.java:243)
- locked <0x00000005c9398570> (a org.apache.dubbo.config.spring.ServiceBean)
at org.apache.dubbo.config.deploy.DefaultModuleDeployer.exportServiceInternal(DefaultModuleDeployer.java:350)
at org.apache.dubbo.config.deploy.DefaultModuleDeployer.exportServices(DefaultModuleDeployer.java:322)
at org.apache.dubbo.config.deploy.DefaultModuleDeployer.startSync(DefaultModuleDeployer.java:158)
- locked <0x00000005c4c594f0> (a org.apache.dubbo.config.deploy.DefaultModuleDeployer)
at org.apache.dubbo.config.deploy.DefaultModuleDeployer.start(DefaultModuleDeployer.java:139)
at org.apache.dubbo.config.spring.context.DubboDeployApplicationListener.onContextRefreshedEvent(DubboDeployApplicationListener.java:113)
at org.apache.dubbo.config.spring.context.DubboDeployApplicationListener.onApplicationEvent(DubboDeployApplicationListener.java:102)
at org.apache.dubbo.config.spring.context.DubboDeployApplicationListener.onApplicationEvent(DubboDeployApplicationListener.java:47)
...
Found 1 deadlock.
- 解决方案
java
@EventListener
public void onDubboStarted(DubboApplicationStateEvent event) {
if (event.getState() == DeployState.STARTED) {
doSomething();
}
}