31集-33集【求助】AIGC返回的对话内容文字转语音失败-《MCU嵌入式AI开发笔记》

31集【求助】AIGC返回的对话内容文字转语音失败-《MCU嵌入式AI开发笔记》

问题描述

ESP32 C3开发板把AIGC大模型返回的对话文字转语音的时候出现错误。

我们先看一下附件Log,

梳理一下程序流程

按键,收到event Event received, cmd:1, data:0x9, data_len:0

也就是audio_event_iface_listen收到了audio codec过来的语音数据。

之后调用baidu_tts_check_event_finish,把tts发送给百度。

之后会调用char *answer = minimax_chat(original_text);,把识别的文字送给大模型。并得到返回的answer。

之后调用baidu_tts_start得到音频流,并发给audio codec。

我们在控制台上输出log信息:

我们看到:

出问题的地方

E (133134) MP3_DE_LIB: pvmp3_framedecoder.cpp:701 (pvmp3_InitDecoder): Memory exhausted

也就是内存分配失败了。

我们需要深入研究一下。又遇到这个问题的么,评论区讨论一下。

32集【求助】AIGC返回的对话内容文字转语音失败2-《MCU嵌入式AI开发笔记》

文字转语音程序流程详解

一个重要的结构体,baidu_tts。用来定义一个pipeline,3个elements。buffer size是:#define DEFAULT_TTS_BUFFER_SIZE (2048)

baidu_tts_init来做tts的初始化。

在这里初始化了i2s_stream_init I2S流,和http_stream_init http流。

初始化mp3_decoder_init mp3流

之后调用

c 复制代码
    audio_pipeline_register(tts->pipeline, tts->http_stream_reader, "tts_http");
    audio_pipeline_register(tts->pipeline, tts->mp3_decoder,        "tts_mp3");
    audio_pipeline_register(tts->pipeline, tts->i2s_writer,         "tts_i2s");
    const char *link_tag[3] = {"tts_http", "tts_mp3", "tts_i2s"};
    audio_pipeline_link(tts->pipeline, &link_tag[0], 3);

把流链接的pipeline上,

i2s_writer
Created with Raphaël 2.3.0 http_stream_reader mp3_decoder i2s_writer

33集

ai_chat_task程序流程


造成内存枯竭的原因是内存分配的太多了,所以需要动态释放一些。

有几种可能:

1、内存分配的太多了,之前的没有释放,需要不断的将之前的释放掉。

2、mp3的element申请的task内存区域太少,导致无法分配(初始化的时候申请了5K)。

3、文字太多,导致语音文件太大,导致无法申请内存。

需要先定位,之后研究一下如何修改程序。

附件

I (79774) AI_CHAT_EXAMPLE: TEMP:31 HUMI:63

I (90994) AI_CHAT_EXAMPLE: TEMP:31 HUMI:64

I (102214) AI_CHAT_EXAMPLE: TEMP:32 HUMI:65

I (113434) AI_CHAT_EXAMPLE: TEMP:32 HUMI:62

I (124654) AI_CHAT_EXAMPLE: TEMP:32 HUMI:57

I (126334) AI_CHAT_EXAMPLE: [ * ] Event received: src_type:1048577, source:0x3fcbf538 cmd:1, data:0x9, data_len:0

W (126334) AUDIO_ELEMENT: [tts_http] Element already stopped

W (126334) AUDIO_ELEMENT: [tts_mp3] Element already stopped

W (126344) AUDIO_ELEMENT: [tts_i2s] Element already stopped

I (126354) AI_CHAT_EXAMPLE: [ * ] Resuming pipeline

I (126354) AUDIO_PIPELINE: Func:audio_pipeline_run, Line:359, MEM Total:53384 Bytes

I (126364) AUDIO_ELEMENT: [vtt_http] AEL_MSG_CMD_RESUME,state:1

I (126374) AUDIO_ELEMENT: [vtt_i2s] AEL_MSG_CMD_RESUME,state:1

I (126374) BAIDU_VTT: [ + ] HTTP client HTTP_STREAM_PRE_REQUEST, lenght=0

I (126384) AUDIO_PIPELINE: Pipeline started

I (126394) AI_CHAT_EXAMPLE: [ * ] Event received: src_type:131072, source:0x3fccd9b8 cmd:8, data:0xe, data_len:4

I (126404) AI_CHAT_EXAMPLE: [ * ] Event received: src_type:131072, source:0x3fcd042c cmd:8, data:0xe, data_len:4

I (126414) AI_CHAT_EXAMPLE: [ * ] Event received: src_type:131072, source:0x3fccd2fc cmd:8, data:0xe, data_len:4

I (126424) AI_CHAT_EXAMPLE: [ * ] TTS Finish

I (126424) AI_CHAT_EXAMPLE: [ * ] Event received: src_type:131072, source:0x3fcd5e78 cmd:8, data:0xc, data_len:4

I (126444) AI_CHAT_EXAMPLE: [ * ] Event received: src_type:131072, source:0x3fcd36bc cmd:10, data:0x0, data_len:0

I (126454) AI_CHAT_EXAMPLE: [ * ] Event received: src_type:131072, source:0x3fcd36Total bytes written: 61440

I (128374) AI_CHAT_EXAMPLE: [ * ] Event received: src_type:1048577, source:0x3fcbfTotal bytes written: 114688

I (130024) AI_CHAT_EXAMPLE: [ * ] Event received: src_type:1048577, source:0x3fcbf538 cmd:4, data:0x9, data_len:0

I (130024) AI_CHAT_EXAMPLE: [ * ] Stop pipeline

W (130034) AUDIO_ELEMENT: IN-[vtt_http] AEL_IO_ABORT

W (130034) HTTP_STREAM: No output due to stopping

I (130044) BAIDU_VTT: [ + ] HTTP client HTTP_STREAM_POST_REQUEST, write end chunked marker

I (131064) BAIDU_VTT: [ + ] HTTP client HTTP_STREAM_FINISH_REQUEST, read_len=141

I (131064) BAIDU_VTT: Got HTTP Response = {"corpus_no":"7401889142447543659","err_msg":"success.","err_no":0,"result":["你能讲好玩的故事吗?"],"sn":"887851329131723386613"}

I (131084) BAIDU_VTT: response_text:你能讲好玩的故事吗?

I (131084) AI_CHAT_EXAMPLE: Original text = 你能讲好玩的故事吗?

I (131604) MINIMAX_CHAT: Need to write 490, written 490

I (132394) MINIMAX_CHAT: read = {"created":1723386615,"model":"abab5.5s-chat","reply":"当然可以啦,你想听什么类型的故事呢?","choices":[{"finish_reason":"stop","messages":[{"sender_type":"BOT","sender_name":"小美","text":"当然可以啦,你想听什么类

型的故事呢?"}]}],"usage":{"total_tokens":82},"input_sensitive":false,"output_sensitive":false,"id":"0307fdf617b71f7a9de4cfdac446a066","base_resp":{"status_code":0,"status_msg":""}}

I (132424) MINIMAX_CHAT: response_text:当然可以啦,你想听什么类型的故事呢?

I (132434) AI_CHAT_EXAMPLE: minimax answer = 当然可以啦,你想听什么类型的故事呢?

I (132444) AUDIO_PIPELINE: Func:audio_pipeline_run, Line:359, MEM Total:52824 Bytes

I (132454) AUDIO_ELEMENT: [tts_http] AEL_MSG_CMD_RESUME,state:1

I (132464) AUDIO_ELEMENT: [tts_mp3] AEL_MSG_CMD_RESUME,state:1

I (132464) MP3_DECODER: MP3 opened

I (132474) AUDIO_ELEMENT: [tts_i2s] AEL_MSG_CMD_RESUME,state:1

I (132464) BAIDU_TTS: [ + ] HTTP client HTTP_STREAM_PRE_REQUEST, lenght=0

I (132484) AUDIO_PIPELINE: Pipeline started

I (132494) AI_CHAT_EXAMPLE: [ * ] Event received: src_type:131072, source:0x3fcd5e78 cmd:11, data:0x3fcb2990, data_len:64

I (132504) AI_CHAT_EXAMPLE: [ * ] Event received: src_type:131072, source:0x3fcd5e78 cmd:8, data:0xe, data_len:4

I (132514) AI_CHAT_EXAMPLE: [ * ] Event received: src_type:131072, source:0x3fcd36bc cmd:11, data:0x3fcb291c, data_len:64

I (132524) AI_CHAT_EXAMPLE: [ * ] Event received: src_type:131072, source:0x3fcd36bc cmd:8, data:0xe, data_len:4

I (132534) AI_CHAT_EXAMPLE: [ * ] Event received: src_type:131072, source:0x3fccd2fc cmd:8, data:0xc, data_len:4

I (133114) HTTP_STREAM: total_bytes=30240

I (133114) AI_CHAT_EXAMPLE: [ * ] Event received: src_type:131072, source:0x3fccd9b8 cmd:10, data:0x0, data_len:0

I (133114) AI_CHAT_EXAMPLE: [ * ] Event received: src_type:131072, source:0x3fccd9b8 cmd:8, data:0xc, data_len:4

I (133134) CODEC_ELEMENT_HELPER: The element is 0x3fcd042c. The reserve data 2 is

0x0.

E (133134) MP3_DE_LIB: pvmp3_framedecoder.cpp:701 (pvmp3_InitDecoder): Memory exhausted

E (133144) MP3_DECODER: Allocate buffer failed. (line 404)

E (133154) AUDIO_ELEMENT: [tts_mp3] AEL_STATUS_ERROR_OPEN,-1

W (133154) AUDIO_ELEMENT: [tts_mp3] audio_element_on_cmd_error,7

I (133164) MP3_DECODER: Closed

I (133154) AI_CHAT_EXAMPLE: [ * ] Event received: src_type:131072, source:0x3fcd042c cmd:8, data:0x1, data_len:4

W (133174) AUDIO_ELEMENT: IN-[tts_i2s] AEL_IO_ABORT

I (133184) AI_CHAT_EXAMPLE: [ * ] Event received: src_type:131072, source:0x3fcd042c cmd:11, data:0x3fcba84c, data_len:64

I (133194) AI_CHAT_EXAMPLE: [ * ] Event received: src_type:131072, source:0x3fccd2fc cmd:11, data:0x3fcba7b4, data_len:64

I (133204) AI_CHAT_EXAMPLE: [ * ] Event received: src_type:131072, source:0x3fccd2fc cmd:8, data:0xe, data_len:4

I (133224) AI_CHAT_EXAMPLE: [ * ] TTS Finish

W (133204) AUDIO_ELEMENT: OUT-[tts_http] AEL_IO_ABORT

I (133234) AI_CHAT_EXAMPLE: [ * ] Event received: src_type:131072, source:0x3fccd9b8 cmd:11, data:0x3fcb8428, data_len:64

I (133244) AI_CHAT_EXAMPLE: [ * ] Event received: src_type:131072, source:0x3fccd9b8 cmd:8, data:0xe, data_len:4

I (135874) AI_CHAT_EXAMPLE: TEMP:32 HUMI:65

I (147094) AI_CHAT_EXAMPLE: TEMP:32 HUMI:71

I (158314) AI_CHAT_EXAMPLE: TEMP:32 HUMI:70

I (169534) AI_CHAT_EXAMPLE: TEMP:32 HUMI:70

I (180754) AI_CHAT_EXAMPLE: TEMP:32 HUMI:69

I (191974) AI_CHAT_EXAMPLE: TEMP:32 HUMI:68

I (203194) AI_CHAT_EXAMPLE: TEMP:32 HUMI:68

相关推荐
YSGZJJ5 分钟前
股指期货的套保策略如何精准选择和规避风险?
人工智能·区块链
无脑敲代码,bug漫天飞7 分钟前
COR 损失函数
人工智能·机器学习
HPC_fac130520678161 小时前
以科学计算为切入点:剖析英伟达服务器过热难题
服务器·人工智能·深度学习·机器学习·计算机视觉·数据挖掘·gpu算力
小陈phd4 小时前
OpenCV从入门到精通实战(九)——基于dlib的疲劳监测 ear计算
人工智能·opencv·计算机视觉
zy张起灵4 小时前
48v72v-100v转12v 10A大功率转换电源方案CSM3100SK
经验分享·嵌入式硬件·硬件工程
冰帝海岸4 小时前
01-spring security认证笔记
java·笔记·spring
Guofu_Liao5 小时前
大语言模型---LoRA简介;LoRA的优势;LoRA训练步骤;总结
人工智能·语言模型·自然语言处理·矩阵·llama
小二·5 小时前
java基础面试题笔记(基础篇)
java·笔记·python
wusong9998 小时前
mongoDB回顾笔记(一)
数据库·笔记·mongodb
猫爪笔记8 小时前
前端:HTML (学习笔记)【1】
前端·笔记·学习·html