31集-33集【求助】AIGC返回的对话内容文字转语音失败-《MCU嵌入式AI开发笔记》

31集【求助】AIGC返回的对话内容文字转语音失败-《MCU嵌入式AI开发笔记》

问题描述

ESP32 C3开发板把AIGC大模型返回的对话文字转语音的时候出现错误。

我们先看一下附件Log,

梳理一下程序流程

按键,收到event Event received, cmd:1, data:0x9, data_len:0

也就是audio_event_iface_listen收到了audio codec过来的语音数据。

之后调用baidu_tts_check_event_finish,把tts发送给百度。

之后会调用char *answer = minimax_chat(original_text);,把识别的文字送给大模型。并得到返回的answer。

之后调用baidu_tts_start得到音频流,并发给audio codec。

我们在控制台上输出log信息:

我们看到:

出问题的地方

E (133134) MP3_DE_LIB: pvmp3_framedecoder.cpp:701 (pvmp3_InitDecoder): Memory exhausted

也就是内存分配失败了。

我们需要深入研究一下。又遇到这个问题的么,评论区讨论一下。

32集【求助】AIGC返回的对话内容文字转语音失败2-《MCU嵌入式AI开发笔记》

文字转语音程序流程详解

一个重要的结构体,baidu_tts。用来定义一个pipeline,3个elements。buffer size是:#define DEFAULT_TTS_BUFFER_SIZE (2048)

baidu_tts_init来做tts的初始化。

在这里初始化了i2s_stream_init I2S流,和http_stream_init http流。

初始化mp3_decoder_init mp3流

之后调用

c 复制代码
    audio_pipeline_register(tts->pipeline, tts->http_stream_reader, "tts_http");
    audio_pipeline_register(tts->pipeline, tts->mp3_decoder,        "tts_mp3");
    audio_pipeline_register(tts->pipeline, tts->i2s_writer,         "tts_i2s");
    const char *link_tag[3] = {"tts_http", "tts_mp3", "tts_i2s"};
    audio_pipeline_link(tts->pipeline, &link_tag[0], 3);

把流链接的pipeline上,

i2s_writer
Created with Raphaël 2.3.0 http_stream_reader mp3_decoder i2s_writer

33集

ai_chat_task程序流程


造成内存枯竭的原因是内存分配的太多了,所以需要动态释放一些。

有几种可能:

1、内存分配的太多了,之前的没有释放,需要不断的将之前的释放掉。

2、mp3的element申请的task内存区域太少,导致无法分配(初始化的时候申请了5K)。

3、文字太多,导致语音文件太大,导致无法申请内存。

需要先定位,之后研究一下如何修改程序。

附件

I (79774) AI_CHAT_EXAMPLE: TEMP:31 HUMI:63

I (90994) AI_CHAT_EXAMPLE: TEMP:31 HUMI:64

I (102214) AI_CHAT_EXAMPLE: TEMP:32 HUMI:65

I (113434) AI_CHAT_EXAMPLE: TEMP:32 HUMI:62

I (124654) AI_CHAT_EXAMPLE: TEMP:32 HUMI:57

I (126334) AI_CHAT_EXAMPLE: [ * ] Event received: src_type:1048577, source:0x3fcbf538 cmd:1, data:0x9, data_len:0

W (126334) AUDIO_ELEMENT: [tts_http] Element already stopped

W (126334) AUDIO_ELEMENT: [tts_mp3] Element already stopped

W (126344) AUDIO_ELEMENT: [tts_i2s] Element already stopped

I (126354) AI_CHAT_EXAMPLE: [ * ] Resuming pipeline

I (126354) AUDIO_PIPELINE: Func:audio_pipeline_run, Line:359, MEM Total:53384 Bytes

I (126364) AUDIO_ELEMENT: [vtt_http] AEL_MSG_CMD_RESUME,state:1

I (126374) AUDIO_ELEMENT: [vtt_i2s] AEL_MSG_CMD_RESUME,state:1

I (126374) BAIDU_VTT: [ + ] HTTP client HTTP_STREAM_PRE_REQUEST, lenght=0

I (126384) AUDIO_PIPELINE: Pipeline started

I (126394) AI_CHAT_EXAMPLE: [ * ] Event received: src_type:131072, source:0x3fccd9b8 cmd:8, data:0xe, data_len:4

I (126404) AI_CHAT_EXAMPLE: [ * ] Event received: src_type:131072, source:0x3fcd042c cmd:8, data:0xe, data_len:4

I (126414) AI_CHAT_EXAMPLE: [ * ] Event received: src_type:131072, source:0x3fccd2fc cmd:8, data:0xe, data_len:4

I (126424) AI_CHAT_EXAMPLE: [ * ] TTS Finish

I (126424) AI_CHAT_EXAMPLE: [ * ] Event received: src_type:131072, source:0x3fcd5e78 cmd:8, data:0xc, data_len:4

I (126444) AI_CHAT_EXAMPLE: [ * ] Event received: src_type:131072, source:0x3fcd36bc cmd:10, data:0x0, data_len:0

I (126454) AI_CHAT_EXAMPLE: [ * ] Event received: src_type:131072, source:0x3fcd36Total bytes written: 61440

I (128374) AI_CHAT_EXAMPLE: [ * ] Event received: src_type:1048577, source:0x3fcbfTotal bytes written: 114688

I (130024) AI_CHAT_EXAMPLE: [ * ] Event received: src_type:1048577, source:0x3fcbf538 cmd:4, data:0x9, data_len:0

I (130024) AI_CHAT_EXAMPLE: [ * ] Stop pipeline

W (130034) AUDIO_ELEMENT: IN-[vtt_http] AEL_IO_ABORT

W (130034) HTTP_STREAM: No output due to stopping

I (130044) BAIDU_VTT: [ + ] HTTP client HTTP_STREAM_POST_REQUEST, write end chunked marker

I (131064) BAIDU_VTT: [ + ] HTTP client HTTP_STREAM_FINISH_REQUEST, read_len=141

I (131064) BAIDU_VTT: Got HTTP Response = {"corpus_no":"7401889142447543659","err_msg":"success.","err_no":0,"result":["你能讲好玩的故事吗?"],"sn":"887851329131723386613"}

I (131084) BAIDU_VTT: response_text:你能讲好玩的故事吗?

I (131084) AI_CHAT_EXAMPLE: Original text = 你能讲好玩的故事吗?

I (131604) MINIMAX_CHAT: Need to write 490, written 490

I (132394) MINIMAX_CHAT: read = {"created":1723386615,"model":"abab5.5s-chat","reply":"当然可以啦,你想听什么类型的故事呢?","choices":[{"finish_reason":"stop","messages":[{"sender_type":"BOT","sender_name":"小美","text":"当然可以啦,你想听什么类

型的故事呢?"}]}],"usage":{"total_tokens":82},"input_sensitive":false,"output_sensitive":false,"id":"0307fdf617b71f7a9de4cfdac446a066","base_resp":{"status_code":0,"status_msg":""}}

I (132424) MINIMAX_CHAT: response_text:当然可以啦,你想听什么类型的故事呢?

I (132434) AI_CHAT_EXAMPLE: minimax answer = 当然可以啦,你想听什么类型的故事呢?

I (132444) AUDIO_PIPELINE: Func:audio_pipeline_run, Line:359, MEM Total:52824 Bytes

I (132454) AUDIO_ELEMENT: [tts_http] AEL_MSG_CMD_RESUME,state:1

I (132464) AUDIO_ELEMENT: [tts_mp3] AEL_MSG_CMD_RESUME,state:1

I (132464) MP3_DECODER: MP3 opened

I (132474) AUDIO_ELEMENT: [tts_i2s] AEL_MSG_CMD_RESUME,state:1

I (132464) BAIDU_TTS: [ + ] HTTP client HTTP_STREAM_PRE_REQUEST, lenght=0

I (132484) AUDIO_PIPELINE: Pipeline started

I (132494) AI_CHAT_EXAMPLE: [ * ] Event received: src_type:131072, source:0x3fcd5e78 cmd:11, data:0x3fcb2990, data_len:64

I (132504) AI_CHAT_EXAMPLE: [ * ] Event received: src_type:131072, source:0x3fcd5e78 cmd:8, data:0xe, data_len:4

I (132514) AI_CHAT_EXAMPLE: [ * ] Event received: src_type:131072, source:0x3fcd36bc cmd:11, data:0x3fcb291c, data_len:64

I (132524) AI_CHAT_EXAMPLE: [ * ] Event received: src_type:131072, source:0x3fcd36bc cmd:8, data:0xe, data_len:4

I (132534) AI_CHAT_EXAMPLE: [ * ] Event received: src_type:131072, source:0x3fccd2fc cmd:8, data:0xc, data_len:4

I (133114) HTTP_STREAM: total_bytes=30240

I (133114) AI_CHAT_EXAMPLE: [ * ] Event received: src_type:131072, source:0x3fccd9b8 cmd:10, data:0x0, data_len:0

I (133114) AI_CHAT_EXAMPLE: [ * ] Event received: src_type:131072, source:0x3fccd9b8 cmd:8, data:0xc, data_len:4

I (133134) CODEC_ELEMENT_HELPER: The element is 0x3fcd042c. The reserve data 2 is

0x0.

E (133134) MP3_DE_LIB: pvmp3_framedecoder.cpp:701 (pvmp3_InitDecoder): Memory exhausted

E (133144) MP3_DECODER: Allocate buffer failed. (line 404)

E (133154) AUDIO_ELEMENT: [tts_mp3] AEL_STATUS_ERROR_OPEN,-1

W (133154) AUDIO_ELEMENT: [tts_mp3] audio_element_on_cmd_error,7

I (133164) MP3_DECODER: Closed

I (133154) AI_CHAT_EXAMPLE: [ * ] Event received: src_type:131072, source:0x3fcd042c cmd:8, data:0x1, data_len:4

W (133174) AUDIO_ELEMENT: IN-[tts_i2s] AEL_IO_ABORT

I (133184) AI_CHAT_EXAMPLE: [ * ] Event received: src_type:131072, source:0x3fcd042c cmd:11, data:0x3fcba84c, data_len:64

I (133194) AI_CHAT_EXAMPLE: [ * ] Event received: src_type:131072, source:0x3fccd2fc cmd:11, data:0x3fcba7b4, data_len:64

I (133204) AI_CHAT_EXAMPLE: [ * ] Event received: src_type:131072, source:0x3fccd2fc cmd:8, data:0xe, data_len:4

I (133224) AI_CHAT_EXAMPLE: [ * ] TTS Finish

W (133204) AUDIO_ELEMENT: OUT-[tts_http] AEL_IO_ABORT

I (133234) AI_CHAT_EXAMPLE: [ * ] Event received: src_type:131072, source:0x3fccd9b8 cmd:11, data:0x3fcb8428, data_len:64

I (133244) AI_CHAT_EXAMPLE: [ * ] Event received: src_type:131072, source:0x3fccd9b8 cmd:8, data:0xe, data_len:4

I (135874) AI_CHAT_EXAMPLE: TEMP:32 HUMI:65

I (147094) AI_CHAT_EXAMPLE: TEMP:32 HUMI:71

I (158314) AI_CHAT_EXAMPLE: TEMP:32 HUMI:70

I (169534) AI_CHAT_EXAMPLE: TEMP:32 HUMI:70

I (180754) AI_CHAT_EXAMPLE: TEMP:32 HUMI:69

I (191974) AI_CHAT_EXAMPLE: TEMP:32 HUMI:68

I (203194) AI_CHAT_EXAMPLE: TEMP:32 HUMI:68

相关推荐
撞南墙者3 分钟前
OpenCV自学系列(1)——简介和GUI特征操作
人工智能·opencv·计算机视觉
OCR_wintone4214 分钟前
易泊车牌识别相机,助力智慧工地建设
人工智能·数码相机·ocr
王哈哈^_^25 分钟前
【数据集】【YOLO】【VOC】目标检测数据集,查找数据集,yolo目标检测算法详细实战训练步骤!
人工智能·深度学习·算法·yolo·目标检测·计算机视觉·pyqt
一者仁心32 分钟前
【AI技术】PaddleSpeech
人工智能
是瑶瑶子啦40 分钟前
【深度学习】论文笔记:空间变换网络(Spatial Transformer Networks)
论文阅读·人工智能·深度学习·视觉检测·空间变换
EasyCVR43 分钟前
萤石设备视频接入平台EasyCVR多品牌摄像机视频平台海康ehome平台(ISUP)接入EasyCVR不在线如何排查?
运维·服务器·网络·人工智能·ffmpeg·音视频
基极向上的三极管1 小时前
【AD】3-4 在原理图中放置元件
嵌入式硬件
徐嵌1 小时前
STM32项目---水质水位检测
stm32·单片机·嵌入式硬件
柳鲲鹏1 小时前
OpenCV视频防抖源码及编译脚本
人工智能·opencv·计算机视觉
西柚小萌新1 小时前
8.机器学习--决策树
人工智能·决策树·机器学习