当前位置：首页 > news >正文

31集-33集【求助】AIGC返回的对话内容文字转语音失败-《MCU嵌入式AI开发笔记》

news 来源：原创 2024/9/21 3:18:38

31集【求助】AIGC返回的对话内容文字转语音失败-《MCU嵌入式AI开发笔记》

问题描述

ESP32 C3开发板把AIGC大模型返回的对话文字转语音的时候出现错误。
我们先看一下附件Log，

梳理一下程序流程

按键，收到event Event received， cmd:1, data:0x9, data_len:0
也就是audio_event_iface_listen收到了audio codec过来的语音数据。
之后调用baidu_tts_check_event_finish，把tts发送给百度。
之后会调用char *answer = minimax_chat(original_text);，把识别的文字送给大模型。并得到返回的answer。
之后调用baidu_tts_start得到音频流，并发给audio codec。
我们在控制台上输出log信息：
在这里插入图片描述
我们看到：

出问题的地方

E (133134) MP3_DE_LIB: pvmp3_framedecoder.cpp:701 (pvmp3_InitDecoder): Memory exhausted
也就是内存分配失败了。
我们需要深入研究一下。又遇到这个问题的么，评论区讨论一下。

32集【求助】AIGC返回的对话内容文字转语音失败2-《MCU嵌入式AI开发笔记》

文字转语音程序流程详解

一个重要的结构体，baidu_tts。用来定义一个pipeline，3个elements。buffer size是：#define DEFAULT_TTS_BUFFER_SIZE (2048)
在这里插入图片描述
baidu_tts_init来做tts的初始化。
在这里初始化了i2s_stream_init I2S流，和http_stream_init http流。
初始化mp3_decoder_init mp3流
之后调用

    audio_pipeline_register(tts->pipeline, tts->http_stream_reader, "tts_http");audio_pipeline_register(tts->pipeline, tts->mp3_decoder,        "tts_mp3");audio_pipeline_register(tts->pipeline, tts->i2s_writer,         "tts_i2s");const char *link_tag[3] = {"tts_http", "tts_mp3", "tts_i2s"};audio_pipeline_link(tts->pipeline, &link_tag[0], 3);

把流链接的pipeline上，
i2s_writer

33集

ai_chat_task程序流程

在这里插入图片描述

造成内存枯竭的原因是内存分配的太多了，所以需要动态释放一些。
有几种可能：
1、内存分配的太多了，之前的没有释放，需要不断的将之前的释放掉。
2、mp3的element申请的task内存区域太少，导致无法分配（初始化的时候申请了5K）。
3、文字太多，导致语音文件太大，导致无法申请内存。

需要先定位，之后研究一下如何修改程序。

附件

I (79774) AI_CHAT_EXAMPLE: TEMP:31 HUMI:63
I (90994) AI_CHAT_EXAMPLE: TEMP:31 HUMI:64
I (102214) AI_CHAT_EXAMPLE: TEMP:32 HUMI:65
I (113434) AI_CHAT_EXAMPLE: TEMP:32 HUMI:62
I (124654) AI_CHAT_EXAMPLE: TEMP:32 HUMI:57
I (126334) AI_CHAT_EXAMPLE: [ * ] Event received: src_type:1048577, source:0x3fcbf538 cmd:1, data:0x9, data_len:0
W (126334) AUDIO_ELEMENT: [tts_http] Element already stopped
W (126334) AUDIO_ELEMENT: [tts_mp3] Element already stopped
W (126344) AUDIO_ELEMENT: [tts_i2s] Element already stopped
I (126354) AI_CHAT_EXAMPLE: [ * ] Resuming pipeline
I (126354) AUDIO_PIPELINE: Func:audio_pipeline_run, Line:359, MEM Total:53384 Bytes

I (126364) AUDIO_ELEMENT: [vtt_http] AEL_MSG_CMD_RESUME,state:1
I (126374) AUDIO_ELEMENT: [vtt_i2s] AEL_MSG_CMD_RESUME,state:1
I (126374) BAIDU_VTT: [ + ] HTTP client HTTP_STREAM_PRE_REQUEST, lenght=0
I (126384) AUDIO_PIPELINE: Pipeline started
I (126394) AI_CHAT_EXAMPLE: [ * ] Event received: src_type:131072, source:0x3fccd9b8 cmd:8, data:0xe, data_len:4
I (126404) AI_CHAT_EXAMPLE: [ * ] Event received: src_type:131072, source:0x3fcd042c cmd:8, data:0xe, data_len:4
I (126414) AI_CHAT_EXAMPLE: [ * ] Event received: src_type:131072, source:0x3fccd2fc cmd:8, data:0xe, data_len:4
I (126424) AI_CHAT_EXAMPLE: [ * ] TTS Finish
I (126424) AI_CHAT_EXAMPLE: [ * ] Event received: src_type:131072, source:0x3fcd5e78 cmd:8, data:0xc, data_len:4
I (126444) AI_CHAT_EXAMPLE: [ * ] Event received: src_type:131072, source:0x3fcd36bc cmd:10, data:0x0, data_len:0
I (126454) AI_CHAT_EXAMPLE: [ * ] Event received: src_type:131072, source:0x3fcd36Total bytes written: 61440
I (128374) AI_CHAT_EXAMPLE: [ * ] Event received: src_type:1048577, source:0x3fcbfTotal bytes written: 114688
I (130024) AI_CHAT_EXAMPLE: [ * ] Event received: src_type:1048577, source:0x3fcbf538 cmd:4, data:0x9, data_len:0
I (130024) AI_CHAT_EXAMPLE: [ * ] Stop pipeline
W (130034) AUDIO_ELEMENT: IN-[vtt_http] AEL_IO_ABORT
W (130034) HTTP_STREAM: No output due to stopping
I (130044) BAIDU_VTT: [ + ] HTTP client HTTP_STREAM_POST_REQUEST, write end chunked marker
I (131064) BAIDU_VTT: [ + ] HTTP client HTTP_STREAM_FINISH_REQUEST, read_len=141
I (131064) BAIDU_VTT: Got HTTP Response = {“corpus_no”:“7401889142447543659”,“err_msg”:“success.”,“err_no”:0,“result”:[“你能讲好玩的故事吗？”],“sn”:“887851329131723386613”}

I (131084) BAIDU_VTT: response_text:你能讲好玩的故事吗？
I (131084) AI_CHAT_EXAMPLE: Original text = 你能讲好玩的故事吗？
I (131604) MINIMAX_CHAT: Need to write 490, written 490
I (132394) MINIMAX_CHAT: read = {“created”:1723386615,“model”:“abab5.5s-chat”,“reply”:“当然可以啦，你想听什么类型的故事呢？”,“choices”:[{“finish_reason”:“stop”,“messages”:[{“sender_type”:“BOT”,“sender_name”:“小美”,“text”:“当然可以啦，你想听什么类
型的故事呢？”}]}],“usage”:{“total_tokens”:82},“input_sensitive”:false,“output_sensitive”:false,“id”:“0307fdf617b71f7a9de4cfdac446a066”,“base_resp”:{“status_code”:0,“status_msg”:“”}}
I (132424) MINIMAX_CHAT: response_text:当然可以啦，你想听什么类型的故事呢？
I (132434) AI_CHAT_EXAMPLE: minimax answer = 当然可以啦，你想听什么类型的故事呢？
I (132444) AUDIO_PIPELINE: Func:audio_pipeline_run, Line:359, MEM Total:52824 Bytes

I (132454) AUDIO_ELEMENT: [tts_http] AEL_MSG_CMD_RESUME,state:1
I (132464) AUDIO_ELEMENT: [tts_mp3] AEL_MSG_CMD_RESUME,state:1
I (132464) MP3_DECODER: MP3 opened
I (132474) AUDIO_ELEMENT: [tts_i2s] AEL_MSG_CMD_RESUME,state:1
I (132464) BAIDU_TTS: [ + ] HTTP client HTTP_STREAM_PRE_REQUEST, lenght=0
I (132484) AUDIO_PIPELINE: Pipeline started
I (132494) AI_CHAT_EXAMPLE: [ * ] Event received: src_type:131072, source:0x3fcd5e78 cmd:11, data:0x3fcb2990, data_len:64
I (132504) AI_CHAT_EXAMPLE: [ * ] Event received: src_type:131072, source:0x3fcd5e78 cmd:8, data:0xe, data_len:4
I (132514) AI_CHAT_EXAMPLE: [ * ] Event received: src_type:131072, source:0x3fcd36bc cmd:11, data:0x3fcb291c, data_len:64
I (132524) AI_CHAT_EXAMPLE: [ * ] Event received: src_type:131072, source:0x3fcd36bc cmd:8, data:0xe, data_len:4
I (132534) AI_CHAT_EXAMPLE: [ * ] Event received: src_type:131072, source:0x3fccd2fc cmd:8, data:0xc, data_len:4
I (133114) HTTP_STREAM: total_bytes=30240
I (133114) AI_CHAT_EXAMPLE: [ * ] Event received: src_type:131072, source:0x3fccd9b8 cmd:10, data:0x0, data_len:0
I (133114) AI_CHAT_EXAMPLE: [ * ] Event received: src_type:131072, source:0x3fccd9b8 cmd:8, data:0xc, data_len:4
I (133134) CODEC_ELEMENT_HELPER: The element is 0x3fcd042c. The reserve data 2 is
0x0.
E (133134) MP3_DE_LIB: pvmp3_framedecoder.cpp:701 (pvmp3_InitDecoder): Memory exhausted
E (133144) MP3_DECODER: Allocate buffer failed. (line 404)
E (133154) AUDIO_ELEMENT: [tts_mp3] AEL_STATUS_ERROR_OPEN,-1
W (133154) AUDIO_ELEMENT: [tts_mp3] audio_element_on_cmd_error,7
I (133164) MP3_DECODER: Closed
I (133154) AI_CHAT_EXAMPLE: [ * ] Event received: src_type:131072, source:0x3fcd042c cmd:8, data:0x1, data_len:4
W (133174) AUDIO_ELEMENT: IN-[tts_i2s] AEL_IO_ABORT
I (133184) AI_CHAT_EXAMPLE: [ * ] Event received: src_type:131072, source:0x3fcd042c cmd:11, data:0x3fcba84c, data_len:64
I (133194) AI_CHAT_EXAMPLE: [ * ] Event received: src_type:131072, source:0x3fccd2fc cmd:11, data:0x3fcba7b4, data_len:64
I (133204) AI_CHAT_EXAMPLE: [ * ] Event received: src_type:131072, source:0x3fccd2fc cmd:8, data:0xe, data_len:4
I (133224) AI_CHAT_EXAMPLE: [ * ] TTS Finish
W (133204) AUDIO_ELEMENT: OUT-[tts_http] AEL_IO_ABORT
I (133234) AI_CHAT_EXAMPLE: [ * ] Event received: src_type:131072, source:0x3fccd9b8 cmd:11, data:0x3fcb8428, data_len:64
I (133244) AI_CHAT_EXAMPLE: [ * ] Event received: src_type:131072, source:0x3fccd9b8 cmd:8, data:0xe, data_len:4
I (135874) AI_CHAT_EXAMPLE: TEMP:32 HUMI:65
I (147094) AI_CHAT_EXAMPLE: TEMP:32 HUMI:71
I (158314) AI_CHAT_EXAMPLE: TEMP:32 HUMI:70
I (169534) AI_CHAT_EXAMPLE: TEMP:32 HUMI:70
I (180754) AI_CHAT_EXAMPLE: TEMP:32 HUMI:69
I (191974) AI_CHAT_EXAMPLE: TEMP:32 HUMI:68
I (203194) AI_CHAT_EXAMPLE: TEMP:32 HUMI:68