SpringCloudStreamkafka接收jsonarray字符串失败

坚持是一种态度2023-10-04 11:44

文章目录

场景现象

  • kafka作为消息队列，作为前端设备数据到后端消费的渠道，也被多个不同微服务消费
  • 一个服务与前端边缘计算设备建立socket消息，接收实时交通事件推送，再将事件发送到kafka里面。此处使用的是Spring Kafka，普通的将事件列表数据转化为字符串后发送
  • 事件信息，需要入库和实时推送，也需要被第三方服务调用，第三方也采取消费kafka的方式
  • 这个服务，作为kafka的消费者，读取事件数据，发送短信给指定用户，但此处使用的是Spring.cloud.stream，函数式编程，绑定kafka的topic
  • 此服务消费时，加了一些打印。但是事件推送到kafka后，看不到任何日志打印，也没有报错。但是使用kafka命令查看，却已经被该消费者消费掉了
  • 消费的源代码如下：
java 复制代码 
    @Bean
    public Consumer<String> boxEventTopicConsumer(HWTrafficEventManager hwTrafficEventManager) {
        log.info("Consumer Received (boxTopic): " + hwTrafficEventManager);
        return value -> {
            log.info("Consumer Received (boxTopic): " + value);
            // todo : do somethings
        };
    }

问题处理

  • 消费某个多分区topic时，发现只有其中一个分区数据有打印，其他都没有，但是查看却发现都消费掉了

    2023-09-21 13:45:13.706 INFO 2184 --- [container-0-C-1] c.newatc.data.kafka.KafkaConfiguration : Consumer Received (boxTopic): {"equipmentNo":0,"laneNum":4,"statisticsCycle":60,"time":1695275100110,"vehicleLanes":[{"averageParkingNum":0.5,"averageSpeed":6.1,"lane":1,"laneFlow":2,"maxQueueLength":12,"nonVehicleAverageTravelTime":0.0,"overStopLineFlow":2,"pedestrianAverageTravelTime":0,"vehicleAverageDelay":15.3,"vehicleAverageTravelTime":0.2},{"averageParkingNum":0.0,"averageSpeed":35.7,"lane":2,"laneFlow":1,"maxQueueLength":0,"nonVehicleAverageTravelTime":0.0,"overStopLineFlow":1,"pedestrianAverageTravelTime":0,"vehicleAverageDelay":0.0,"vehicleAverageTravelTime":0.0},{"averageParkingNum":0.0,"averageSpeed":0.0,"lane":3,"laneFlow":3,"maxQueueLength":0,"nonVehicleAverageTravelTime":0.0,"overStopLineFlow":0,"pedestrianAverageTravelTime":0,"vehicleAverageDelay":0.0,"vehicleAverageTravelTime":0.0},{"averageParkingNum":0.0,"averageSpeed":25.4,"lane":4,"laneFlow":1,"maxQueueLength":0,"nonVehicleAverageTravelTime":0.0,"overStopLineFlow":5,"pedestrianAverageTravelTime":0,"vehicleAverageDelay":0.0,"vehicleAverageTravelTime":1.6}]}

  • 最终测试发现，列表转化为jsonarray字符串，无法打印。但是单条数据json字符串却可以

  • 将代码改为String[]后发现报错，代码如下

java 复制代码 
    @Bean
    public Consumer<String[]> boxEventTopicConsumer(HWTrafficEventManager hwTrafficEventManager) {
        log.info("Consumer Received (boxTopic): " + hwTrafficEventManager);
        return value -> {
            log.info("Consumer Received (boxTopic): " + value.length);

        };
    }

  • 报错如下：

    2023-09-21 11:47:59.684 ERROR 9112 --- [container-0-C-1] o.s.kafka.listener.DefaultErrorHandler : Backoff none exhausted for boxTopic-3@29941

    org.springframework.kafka.listener.ListenerExecutionFailedException: Listener failed; nested exception is org.springframework.messaging.converter.MessageConversionException: Could not read JSON: Cannot deserialize value of type [Ljava.lang.String; from Object value (token JsonToken.START_OBJECT)
    at [Source: (byte[])"[{"equipmentNo":0,"eventCode":8,"eventType":1,"laneTurn":0,"relationNo":1,"time":1695267981221}]"; line: 1, column: 2] (through reference chain: java.lang.Object[][0]); nested exception is com.fasterxml.jackson.databind.exc.MismatchedInputException: Cannot deserialize value of type [Ljava.lang.String; from Object value (token JsonToken.START_OBJECT)
    at [Source: (byte[])"[{"equipmentNo":0,"eventCode":8,"eventType":1,"laneTurn":0,"relationNo":1,"time":1695267981221}]"; line: 1, column: 2] (through reference chain: java.lang.Object[][0]), failedMessage=GenericMessage [payload=byte[96], headers={kafka_offset=29941, kafka_consumer=org.apache.kafka.clients.consumer.KafkaConsumer@3c31888e, deliveryAttempt=3, kafka_timestampType=CREATE_TIME, kafka_receivedPartitionId=3, kafka_receivedMessageKey=[B@3fcf7bd1, kafka_receivedTopic=boxTopic, kafka_receivedTimestamp=1695267871945, contentType=application/json, kafka_groupId=data-center}]
    at org.springframework.kafka.listener.KafkaMessageListenerContainerListenerConsumer.decorateException(KafkaMessageListenerContainer.java:2683) at org.springframework.kafka.listener.KafkaMessageListenerContainerListenerConsumer.doInvokeOnMessage(KafkaMessageListenerContainer.java:2649)
    at org.springframework.kafka.listener.KafkaMessageListenerContainerListenerConsumer.invokeOnMessage(KafkaMessageListenerContainer.java:2609) at org.springframework.kafka.listener.KafkaMessageListenerContainerListenerConsumer.doInvokeRecordListener(KafkaMessageListenerContainer.java:2536)
    at org.springframework.kafka.listener.KafkaMessageListenerContainerListenerConsumer.doInvokeWithRecords(KafkaMessageListenerContainer.java:2427) at org.springframework.kafka.listener.KafkaMessageListenerContainerListenerConsumer.invokeRecordListener(KafkaMessageListenerContainer.java:2305)
    at org.springframework.kafka.listener.KafkaMessageListenerContainerListenerConsumer.invokeListener(KafkaMessageListenerContainer.java:1979) at org.springframework.kafka.listener.KafkaMessageListenerContainerListenerConsumer.invokeIfHaveRecords(KafkaMessageListenerContainer.java:1364)
    at org.springframework.kafka.listener.KafkaMessageListenerContainerListenerConsumer.pollAndInvoke(KafkaMessageListenerContainer.java:1355) at org.springframework.kafka.listener.KafkaMessageListenerContainerListenerConsumer.run(KafkaMessageListenerContainer.java:1247)
    at java.base/java.util.concurrent.ExecutorsRunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.run$$capture(FutureTask.java:264)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java)
    at java.base/java.lang.Thread.run(Thread.java:834)
    Caused by: org.springframework.messaging.converter.MessageConversionException: Could not read JSON: Cannot deserialize value of type [Ljava.lang.String; from Object value (token JsonToken.START_OBJECT)
    at [Source: (byte[])"[{"equipmentNo":0,"eventCode":8,"eventType":1,"laneTurn":0,"relationNo":1,"time":1695267981221}]"; line: 1, column: 2] (through reference chain: java.lang.Object[][0]); nested exception is com.fasterxml.jackson.databind.exc.MismatchedInputException: Cannot deserialize value of type [Ljava.lang.String; from Object value (token JsonToken.START_OBJECT)
    at [Source: (byte[])"[{"equipmentNo":0,"eventCode":8,"eventType":1,"laneTurn":0,"relationNo":1,"time":1695267981221}]"; line: 1, column: 2] (through reference chain: java.lang.Object[][0])
    at org.springframework.messaging.converter.MappingJackson2MessageConverter.convertFromInternal(MappingJackson2MessageConverter.java:237)
    at org.springframework.cloud.stream.converter.ApplicationJsonMessageMarshallingConverter.convertFromInternal(ApplicationJsonMessageMarshallingConverter.java:113)
    at org.springframework.messaging.converter.AbstractMessageConverter.fromMessage(AbstractMessageConverter.java:185)
    at org.springframework.messaging.converter.AbstractMessageConverter.fromMessage(AbstractMessageConverter.java:176)
    at org.springframework.cloud.function.context.config.SmartCompositeMessageConverter.fromMessage(SmartCompositeMessageConverter.java:48)
    at org.springframework.cloud.function.context.catalog.SimpleFunctionRegistryFunctionInvocationWrapper.convertInputMessageIfNecessary(SimpleFunctionRegistry.java:1282) at org.springframework.cloud.function.context.catalog.SimpleFunctionRegistryFunctionInvocationWrapper.convertInputIfNecessary(SimpleFunctionRegistry.java:1057)
    at org.springframework.cloud.function.context.catalog.SimpleFunctionRegistryFunctionInvocationWrapper.doApply(SimpleFunctionRegistry.java:696) at org.springframework.cloud.function.context.catalog.SimpleFunctionRegistryFunctionInvocationWrapper.apply(SimpleFunctionRegistry.java:551)
    at org.springframework.cloud.stream.function.PartitionAwareFunctionWrapper.apply(PartitionAwareFunctionWrapper.java:84)
    at org.springframework.cloud.stream.function.FunctionConfigurationFunctionWrapper.apply(FunctionConfiguration.java:754) at org.springframework.cloud.stream.function.FunctionConfigurationFunctionToDestinationBinder1.handleMessageInternal(FunctionConfiguration.java:586) at org.springframework.integration.handler.AbstractMessageHandler.handleMessage(AbstractMessageHandler.java:56) at org.springframework.integration.dispatcher.AbstractDispatcher.tryOptimizedDispatch(AbstractDispatcher.java:115) at org.springframework.integration.dispatcher.UnicastingDispatcher.doDispatch(UnicastingDispatcher.java:133) at org.springframework.integration.dispatcher.UnicastingDispatcher.dispatch(UnicastingDispatcher.java:106) at org.springframework.integration.channel.AbstractSubscribableChannel.doSend(AbstractSubscribableChannel.java:72) at org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:317) at org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:272) at org.springframework.messaging.core.GenericMessagingTemplate.doSend(GenericMessagingTemplate.java:187) at org.springframework.messaging.core.GenericMessagingTemplate.doSend(GenericMessagingTemplate.java:166) at org.springframework.messaging.core.GenericMessagingTemplate.doSend(GenericMessagingTemplate.java:47) at org.springframework.messaging.core.AbstractMessageSendingTemplate.send(AbstractMessageSendingTemplate.java:109) at org.springframework.integration.endpoint.MessageProducerSupport.sendMessage(MessageProducerSupport.java:216) at org.springframework.integration.kafka.inbound.KafkaMessageDrivenChannelAdapter.sendMessageIfAny(KafkaMessageDrivenChannelAdapter.java:397) at org.springframework.integration.kafka.inbound.KafkaMessageDrivenChannelAdapter.access300(KafkaMessageDrivenChannelAdapter.java:83)
    at org.springframework.integration.kafka.inbound.KafkaMessageDrivenChannelAdapterIntegrationRecordMessageListener.onMessage(KafkaMessageDrivenChannelAdapter.java:454) at org.springframework.integration.kafka.inbound.KafkaMessageDrivenChannelAdapterIntegrationRecordMessageListener.onMessage(KafkaMessageDrivenChannelAdapter.java:428)
    at org.springframework.kafka.listener.adapter.RetryingMessageListenerAdapter.lambdaonMessage0(RetryingMessageListenerAdapter.java:125)
    at org.springframework.retry.support.RetryTemplate.doExecute(RetryTemplate.java:329)
    at org.springframework.retry.support.RetryTemplate.execute(RetryTemplate.java:255)
    at org.springframework.kafka.listener.adapter.RetryingMessageListenerAdapter.onMessage(RetryingMessageListenerAdapter.java:119)
    at org.springframework.kafka.listener.adapter.RetryingMessageListenerAdapter.onMessage(RetryingMessageListenerAdapter.java:42)
    at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.doInvokeOnMessage(KafkaMessageListenerContainer.java:2629)
    ... 12 common frames omitted
    Caused by: com.fasterxml.jackson.databind.exc.MismatchedInputException: Cannot deserialize value of type [Ljava.lang.String; from Object value (token JsonToken.START_OBJECT)
    at [Source: (byte[])"[{"equipmentNo":0,"eventCode":8,"eventType":1,"laneTurn":0,"relationNo":1,"time":1695267981221}]"; line: 1, column: 2] (through reference chain: java.lang.Object[][0])
    at com.fasterxml.jackson.databind.exc.MismatchedInputException.from(MismatchedInputException.java:59)
    at com.fasterxml.jackson.databind.DeserializationContext.reportInputMismatch(DeserializationContext.java:1741)
    at com.fasterxml.jackson.databind.DeserializationContext.handleUnexpectedToken(DeserializationContext.java:1515)
    at com.fasterxml.jackson.databind.DeserializationContext.handleUnexpectedToken(DeserializationContext.java:1420)
    at com.fasterxml.jackson.databind.DeserializationContext.extractScalarFromObject(DeserializationContext.java:932)
    at com.fasterxml.jackson.databind.deser.std.StdDeserializer._parseString(StdDeserializer.java:1292)
    at com.fasterxml.jackson.databind.deser.std.StringArrayDeserializer.deserialize(StringArrayDeserializer.java:166)
    at com.fasterxml.jackson.databind.deser.std.StringArrayDeserializer.deserialize(StringArrayDeserializer.java:25)
    at com.fasterxml.jackson.databind.deser.DefaultDeserializationContext.readRootValue(DefaultDeserializationContext.java:322)
    at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:4674)
    at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3723)
    at org.springframework.messaging.converter.MappingJackson2MessageConverter.convertFromInternal(MappingJackson2MessageConverter.java:223)
    ... 45 common frames omitted

  • 通过日志可以看出，是在接收数据的解析时发生了问题，jsonarray的字符串无法接受解析

  • 在消息发生时，在正常消息后面，加一个"-"，可以正常接收了，就是解析时需要先删掉尾字符再转json

  • 后面又发现，如果是作为字符接收，再转为字符串，是可以的

相关推荐
小马爱打代码33 分钟前
Kafka - 消息零丢失实战
分布式·kafka
长河35 分钟前
Kafka系列教程 - Kafka 运维 -8
运维·分布式·kafka
暮乘白帝过重山1 小时前
Singleton和Prototype的作用域与饿汉式/懒汉式的初始化方式
spring·原型模式·prototype·饿汉式·singleton·懒汉式
ejinxian2 小时前
Spring AI Alibaba 快速开发生成式 Java AI 应用
java·人工智能·spring
杉之2 小时前
SpringBlade 数据库字段的自动填充
java·笔记·学习·spring·tomcat
圈圈编码2 小时前
Spring Task 定时任务
java·前端·spring
爱的叹息3 小时前
Java 连接 Redis 的驱动（Jedis、Lettuce、Redisson、Spring Data Redis）分类及对比
java·redis·spring
松韬3 小时前
Spring + Redisson：从 0 到 1 搭建高可用分布式缓存系统
java·redis·分布式·spring·缓存
天上掉下来个程小白4 小时前
Redis-14.在Java中操作Redis-Spring Data Redis使用方式-操作列表类型的数据
java·redis·spring·springboot·苍穹外卖
汤姆大聪明4 小时前
Redisson 操作 Redis Stream 消息队列详解及实战案例
redis·spring·缓存·maven
热门推荐
01我决定放弃搞 Java 了02DeepSeek各版本说明与优缺点分析03如何在WPS和Word/Excel中直接使用DeepSeek功能04RAG 实践- Ollama+RagFlow 部署本地知识库05苍穹外卖面试总结06从零安装 LLaMA-Factory 微调 Qwen 大模型成功及所有的坑07本地化部署AI知识库：基于Ollama+DeepSeek+AnythingLLM保姆级教程08DeepSeek R1本地化部署 Ollama + Chatbox 打造最强 AI 工具09如何本地部署AI智能体平台，带你手搓一个AI Agent10Android下HWC以及drm_hwcomposer普法(上)