ESP32-CAM视觉语音AI助手（千问平台）：摄像头、INMP441、MAX98357与单片机数据接入

安生生申2026-10-14 10:41

本项目使用AI-Thinker ESP32-CAM、INMP441数字麦克风和MAX98357 I2S功放，实现了一套可以"看、听、说"的嵌入式AI助手。单片机还可以通过ESP32-CAM的下载串口上传温度、湿度、烟雾等设备数据，让AI在回答问题时同时参考现场图片、用户语音和传感器状态。

项目支持两种主要工作模式：

  • PZ：拍摄图片、录制语音、调用多模态AI并播放语音回答。

  • LY：只录制语音、调用AI并播放语音回答。

此外还提供独立的麦克风与功放自检、静音自动停止录音、Wi-Fi自动重连、TTS错误提示和单片机数据缓存功能。

一、实现的功能

1. 拍照＋语音多模态问答

单片机或串口助手发送：PZ\r\n

ESP32-CAM依次完成：

  1. 初始化摄像头并打开闪光灯。

  2. 拍摄JPEG图片并复制到PSRAM。

  3. 释放摄像头和I2S0资源。

  4. 通过INMP441录制用户语音。

  5. 将图片、WAV音频和最近一次设备数据组装成请求。

  6. 调用多模态模型获得中文回答。

  7. 通过UART0返回文字答案。

  8. 调用TTS生成WAV语音。

  9. 通过MAX98357和喇叭播放回答。

2. 纯语音问答

发送：LY\r\n

程序跳过拍照，直接录音并调用AI。该模式适合询问温度、湿度、设备状态等不需要图像的问题。

3. 单片机设备数据上传

除PZ、LY、SPK、MIC外，其他完整串口行都会被保存为最新设备数据。例如：

复制代码 
DATA:TEMP=25.3C,HUMI=60%,SMOKE=12\r\n

随后发送LY或PZ，用户可以直接说"现在温度是多少"，AI会同时参考TEMP=25.3C进行回答。

程序目前只保存最近一行设备数据，因此建议把本次需要的全部传感器值组合在同一行中发送。

4. 语音活动检测与静音结束

录音并不是无条件等待10秒，而是加入了简单的自适应VAD：

  • 开始后的前400ms估算环境底噪。

  • 检测到连续语音后返回VOICE:START。

  • 用户说完并连续静音1.2秒后自动停止。

  • 一直不说话时，约4秒后停止并返回ERR:no speech。

  • 用户持续说话时，最长录制10秒。

5.设备自检查指令

麦克风自检发送：MIC\r\n

程序采集约2秒音频并返回平均幅度和峰值：

TEST:MIC_START TEST:MIC,AVG=123,PEAK=4567

对着麦克风说话或拍手时，数值应明显变化。

功放和喇叭自检

发送：SPK\r\n

程序不访问网络，直接生成约2秒的1kHz测试音：

TEST:SPK_START TEST:SPK_OK

该功能可以把"功放接线问题"和"TTS网络问题"分开排查。

二.硬件说明

  • AI-Thinker ESP32-CAM，带PSRAM。

  • INMP441 I2S数字麦克风模块。

  • MAX98357 I2S数字功放模块。

  • 4Ω或8Ω扬声器。

三.环境说明

开发板：AI Thinker ESP32-CAM ESP32板包：2.0.16或兼容的2.0.x版本 Upload Speed：115200 Flash Mode：DIO Flash Frequency：40MHz Partition Scheme：Huge APP (3MB No OTA/1MB SPIFFS) PSRAM：Enabled 串口：115200，8N1

程序使用旧版driver/i2s.h接口。如果升级到ESP32 Arduino Core 3.x，I2S接口可能需要适配，因此没有必要时建议继续使用已经验证的2.0.16。

需要安装的第三方库：

复制代码 
ArduinoJson 6.x

以下库由ESP32板包提供：

复制代码 
#include "esp_camera.h"
#include <WiFi.h>
#include <WiFiClientSecure.h>
#include <HTTPClient.h>
#include "driver/i2s.h"

四.代码

cpp 复制代码 
/*
 * ============================================================
 *  ESP32-CAM 语音/拍照 智能问答 (千问 Qwen3.8-Omni-Flash)
 * ------------------------------------------------------------
 *  串口只做两件事:
 *    1) 发指令:  "PZ"  -> 拍照 + 录音识别
 *               "LY"  -> 只录音, 语音识别
 *               "SPK" -> 本地功放/喇叭测试音
 *               "MIC" -> 本地麦克风幅度测试
 *    2) 发单片机数据: 其它任意一行(如 "TEMP:25.3") 作为上下文缓存
 *  问题都用"语音"说, 串口不传问题。
 *  两种指令都录音 + 语音播报(TTS)。
 *
 *  关键: 摄像头和 I2S 音频都用 I2S0, 不能同时占用。
 *        所以拍照时才初始化摄像头, 拍完立即 deinit 释放, 再录音。
 *
 *  硬件: ESP32-CAM(AI-Thinker/PSRAM) + INMP441 + MAX98357
 *  引脚:
 *    I2S BCLK = IO14    I2S WS = IO15
 *    I2S 麦SD(进) = IO2  I2S 功放DIN(出) = IO12  功放SD/EN = IO13
 *    摄像头闪光灯 = IO4
 *    串口: IO3/U0R(RX)、IO1/U0T(TX)，下载口与单片机通信分时复用
 *
 *  UART0 注意: CH340 TX 与单片机 TX 禁止同时连接 IO3，必须用跳帽/开关二选一。
 *  音频注意: IO14 已从原 UART2 释放，专门用作 I2S BCLK。
 *  注意: IO12 是启动脚, 接功放 DIN 时建议加 10k 下拉到 GND。
 *  依赖: ArduinoJson, ESP32 官方板包
 * ============================================================
 */

#include "esp_camera.h"
#include <WiFi.h>
#include <WiFiClientSecure.h>
#include <HTTPClient.h>
#include <ArduinoJson.h>
#include "driver/i2s.h"

// ============================================================
//                       用户配置区
// ============================================================
#define WIFI_SSID        "12345678"     // 必须 2.4G
#define WIFI_PASS        "12345678"

#define API_KEY    ""
#define API_HOST   "maas.qianwenaiapi.com"

// 看懂图片/听懂说话 的全模态模型
#define OMNI_MODEL "qwen3.8-omni-flash"
#define OMNI_PATH  "/compatible-mode/v1/chat/completions"

// 回答指令(会连同设备数据一起发给模型)
#define ASK_PROMPT "请结合提供的音频(和图片)以及设备数据, 用一句简体中文回答。"

// TTS 播报(最便宜)
#define ENABLE_TTS   1
#define TTS_MODEL    "qwen3-tts-flash"
#define TTS_PATH     "/api/v1/services/aigc/multimodal-generation/generation"
#define TTS_VOICE    "Cherry"
#define TTS_RATE     16000

// 录音 / 图像
#define SAMPLE_RATE      16000
#define RECORD_SECONDS   10
#define FRAME_SIZE       FRAMESIZE_VGA
#define JPEG_QUALITY     12

// 录音静音检测：最多录10秒，说完静音1.2秒结束；一直不说话4秒退出
#define VAD_CALIBRATE_MS      400
#define VAD_START_TIMEOUT_MS  4000
#define VAD_START_CONFIRM_MS  80
#define VAD_SILENCE_MS        1200
#define VAD_MIN_LEVEL         120

// 闪光灯电平(高电平点亮)
#define FLASH_ON         HIGH
#define FLASH_OFF        LOW

// I2S 引脚
#define I2S_BCLK_PIN     14
#define I2S_WS_PIN       15
#define I2S_MIC_SD_PIN   2
#define I2S_AMP_DIN_PIN  12
#define I2S_AMP_SD_PIN   13

// 下载口 UART0 与单片机通信：U0R=GPIO3(RX)，U0T=GPIO1(TX)
#define MCU_BAUD          115200
// 设为 1 会把调试日志也发到 UART0；正式连接单片机时保持为 0
#define UART_DEBUG        0
// ============================================================

#if UART_DEBUG
  #define DBG_PRINT(...)   Serial.print(__VA_ARGS__)
  #define DBG_PRINTLN(...) Serial.println(__VA_ARGS__)
  #define DBG_PRINTF(...)  Serial.printf(__VA_ARGS__)
#else
  #define DBG_PRINT(...)   do {} while (0)
  #define DBG_PRINTLN(...) do {} while (0)
  #define DBG_PRINTF(...)  do {} while (0)
#endif

#define CAMERA_MODEL_AI_THINKER
#include "camera_pins.h"

String rxLine;
String g_sensorData;

// ------------------- Base64 -------------------
static const char B64[] =
    "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/";
static size_t base64Encode(const uint8_t *s, size_t n, char *d) {
  size_t o = 0;
  for (size_t i = 0; i < n; i += 3) {
    uint32_t v = (uint32_t)s[i] << 16;
    if (i + 1 < n) v |= (uint32_t)s[i + 1] << 8;
    if (i + 2 < n) v |= (uint32_t)s[i + 2];
    d[o++] = B64[(v >> 18) & 0x3F];
    d[o++] = B64[(v >> 12) & 0x3F];
    d[o++] = (i + 1 < n) ? B64[(v >> 6) & 0x3F] : '=';
    d[o++] = (i + 2 < n) ? B64[v & 0x3F] : '=';
  }
  return o;
}

static String jsonEscape(const String &s) {
  String o; o.reserve(s.length() + 8);
  for (size_t i = 0; i < s.length(); i++) {
    char c = s[i];
    if (c == '"') o += "\\\"";
    else if (c == '\\') o += "\\\\";
    else if (c == '\n') o += "\\n";
    else if (c == '\r') o += "\\r";
    else if (c == '\t') o += "\\t";
    else o += c;
  }
  return o;
}

static void writeWavHeader(uint8_t *h, uint32_t dataLen, uint32_t rate, uint16_t ch, uint16_t bits) {
  uint32_t byteRate = rate * ch * bits / 8;
  uint16_t blockAlign = ch * bits / 8;
  memcpy(h + 0, "RIFF", 4);
  uint32_t chunk = 36 + dataLen; memcpy(h + 4, &chunk, 4);
  memcpy(h + 8, "WAVE", 4);
  memcpy(h + 12, "fmt ", 4);
  uint32_t sz = 16; memcpy(h + 16, &sz, 4);
  uint16_t fmt = 1; memcpy(h + 20, &fmt, 2);
  memcpy(h + 22, &ch, 2);
  memcpy(h + 24, &rate, 4);
  memcpy(h + 28, &byteRate, 4);
  memcpy(h + 32, &blockAlign, 2);
  memcpy(h + 34, &bits, 2);
  memcpy(h + 36, "data", 4);
  memcpy(h + 40, &dataLen, 4);
}

// ------------------- 摄像头(按需, 用完 deinit) -------------------
static bool initCamera() {
  camera_config_t c = {};
  c.ledc_channel = LEDC_CHANNEL_0; c.ledc_timer = LEDC_TIMER_0;
  c.pin_d0 = Y2_GPIO_NUM; c.pin_d1 = Y3_GPIO_NUM; c.pin_d2 = Y4_GPIO_NUM; c.pin_d3 = Y5_GPIO_NUM;
  c.pin_d4 = Y6_GPIO_NUM; c.pin_d5 = Y7_GPIO_NUM; c.pin_d6 = Y8_GPIO_NUM; c.pin_d7 = Y9_GPIO_NUM;
  c.pin_xclk = XCLK_GPIO_NUM; c.pin_pclk = PCLK_GPIO_NUM; c.pin_vsync = VSYNC_GPIO_NUM;
  c.pin_href = HREF_GPIO_NUM; c.pin_sccb_sda = SIOD_GPIO_NUM; c.pin_sccb_scl = SIOC_GPIO_NUM;
  c.pin_pwdn = PWDN_GPIO_NUM; c.pin_reset = RESET_GPIO_NUM;
  c.xclk_freq_hz = 20000000;
  c.pixel_format = PIXFORMAT_JPEG;
  c.frame_size = FRAME_SIZE;
  c.jpeg_quality = JPEG_QUALITY;
  c.fb_count = 1;
  c.fb_location = CAMERA_FB_IN_PSRAM;
  c.grab_mode = CAMERA_GRAB_WHEN_EMPTY;
  if (!psramFound()) { c.frame_size = FRAMESIZE_SVGA; c.fb_location = CAMERA_FB_IN_DRAM; }
  return esp_camera_init(&c) == ESP_OK;
}

// 拍照: 初始化->拍->拷出->deinit(释放 I2S0)。成功返回拷贝的 JPEG 缓冲(需 free)
static uint8_t *captureJpeg(size_t *outLen) {
  if (!initCamera()) { DBG_PRINTLN("[CAM] init fail"); return nullptr; }
  digitalWrite(LED_GPIO_NUM, FLASH_ON); delay(120);
  camera_fb_t *fb = nullptr;
  for (int i = 0; i < 3 && !fb; i++) { fb = esp_camera_fb_get(); if (!fb) delay(200); }
  digitalWrite(LED_GPIO_NUM, FLASH_OFF);

  uint8_t *copy = nullptr;
  if (fb && fb->format == PIXFORMAT_JPEG && fb->len > 0) {
    copy = (uint8_t *)ps_malloc(fb->len);
    if (!copy) copy = (uint8_t *)malloc(fb->len);
    if (copy) { memcpy(copy, fb->buf, fb->len); *outLen = fb->len; }
  }
  if (fb) esp_camera_fb_return(fb);
  esp_camera_deinit();     // 释放 I2S0 给录音用
  return copy;
}

// ------------------- 功放静音 -------------------
static void ampMute() {
  pinMode(I2S_AMP_SD_PIN, OUTPUT);
  digitalWrite(I2S_AMP_SD_PIN, LOW);
  pinMode(I2S_AMP_DIN_PIN, OUTPUT);
  digitalWrite(I2S_AMP_DIN_PIN, LOW);
}

static void ampEnable() {
  pinMode(I2S_AMP_SD_PIN, OUTPUT);
  digitalWrite(I2S_AMP_SD_PIN, HIGH);
  delay(5);
}

// ------------------- I2S 录音 (I2S0, 摄像头已 deinit) -------------------
static bool i2sRxInstall() {
  i2s_config_t cfg = {};
  cfg.mode = (i2s_mode_t)(I2S_MODE_MASTER | I2S_MODE_RX);
  cfg.sample_rate = SAMPLE_RATE;
  cfg.bits_per_sample = I2S_BITS_PER_SAMPLE_32BIT;
  cfg.channel_format = I2S_CHANNEL_FMT_ONLY_LEFT;
  cfg.communication_format = I2S_COMM_FORMAT_STAND_I2S;
  cfg.intr_alloc_flags = ESP_INTR_FLAG_LEVEL1;
  cfg.dma_buf_count = 8; cfg.dma_buf_len = 128;
  if (i2s_driver_install(I2S_NUM_0, &cfg, 0, NULL) != ESP_OK) { DBG_PRINTLN("[i2s] install fail"); return false; }
  i2s_pin_config_t p = {};
  p.bck_io_num = I2S_BCLK_PIN; p.ws_io_num = I2S_WS_PIN;
  p.data_out_num = I2S_PIN_NO_CHANGE; p.data_in_num = I2S_MIC_SD_PIN;
  i2s_set_pin(I2S_NUM_0, &p);
  i2s_zero_dma_buffer(I2S_NUM_0);
  ampMute();
  return true;
}

static uint8_t *recordPcm(Stream &out, size_t *outLen, bool *speechDetected) {
  const size_t total = (size_t)SAMPLE_RATE * RECORD_SECONDS;
  *speechDetected = false;
  uint8_t *pcm = (uint8_t *)ps_malloc(total * 2);
  if (!pcm) pcm = (uint8_t *)malloc(total * 2);
  if (!pcm) { out.print("[i2s] pcm alloc fail\r\n"); return nullptr; }
  if (!i2sRxInstall()) { free(pcm); out.print("ERR:i2s install\r\n"); return nullptr; }

  static uint32_t raw[128];   // 放静态, 省栈
  int16_t *buf16 = (int16_t *)pcm;
  size_t got = 0; int lastSec = -1;
  uint32_t lastData = millis();
  uint32_t recordStart = lastData;
  uint32_t voiceCandidate = 0;
  uint32_t lastVoice = 0;
  uint32_t noiseFloor = 0xFFFFFFFFU;
  const char *endReason = "MAX";
  out.print("[i2s] start read\r\n");
  while (got < total) {
    size_t br = 0;
    i2s_read(I2S_NUM_0, raw, sizeof(raw), &br, 100);
    size_t n = br / 4;
    if (n) lastData = millis();
    uint64_t blockAbs = 0;
    uint32_t blockPeak = 0;
    size_t blockSamples = 0;
    for (size_t i = 0; i < n && got < total; i++) {
      int16_t sample = (int16_t)(raw[i] >> 14);
      buf16[got++] = sample;
      int32_t signedLevel = sample;
      uint32_t level = signedLevel < 0 ? (uint32_t)(-signedLevel) : (uint32_t)signedLevel;
      blockAbs += level;
      if (level > blockPeak) blockPeak = level;
      blockSamples++;
    }

    uint32_t now = millis();
    if (blockSamples > 0) {
      uint32_t blockAvg = (uint32_t)(blockAbs / blockSamples);
      if (now - recordStart < VAD_CALIBRATE_MS) {
        if (blockAvg < noiseFloor) noiseFloor = blockAvg;
      } else {
        if (noiseFloor == 0xFFFFFFFFU) noiseFloor = 0;
        uint32_t threshold = noiseFloor * 3U + 40U;
        if (threshold < VAD_MIN_LEVEL) threshold = VAD_MIN_LEVEL;
        bool voiced = blockAvg > threshold || blockPeak > threshold * 4U;

        if (!*speechDetected) {
          if (voiced) {
            if (voiceCandidate == 0) voiceCandidate = now;
            if (now - voiceCandidate >= VAD_START_CONFIRM_MS) {
              *speechDetected = true;
              lastVoice = now;
              out.print("VOICE:START\r\n");
            }
          } else {
            voiceCandidate = 0;
            noiseFloor = (noiseFloor * 15U + blockAvg) / 16U;
          }

          if (!*speechDetected && now - recordStart >= VAD_START_TIMEOUT_MS) {
            endReason = "NO_SPEECH";
            break;
          }
        } else {
          if (voiced) lastVoice = now;
          if (now - lastVoice >= VAD_SILENCE_MS) {
            endReason = "SILENCE";
            break;
          }
        }
      }
    }

    int sec = (int)(got / SAMPLE_RATE);
    if (sec != lastSec) {
      lastSec = sec;
      out.print("REC "); out.print(sec); out.print("/"); out.print(RECORD_SECONDS); out.print("s\r\n");
    }
    if (millis() - lastData > 5000) { out.print("ERR:mic no data\r\n"); break; }
    yield();
  }
  i2s_driver_uninstall(I2S_NUM_0);
  ampMute();
  *outLen = got * 2;
  out.print("REC:END,"); out.print(endReason); out.print("\r\n");
  return pcm;
}

// ------------------- I2S 播放 (I2S0) -------------------
static bool playWav(uint8_t *wav, size_t len, String &err) {
  if (len < 44 || memcmp(wav, "RIFF", 4) != 0 || memcmp(wav + 8, "WAVE", 4) != 0) {
    err = "not wav";
    return false;
  }

  uint16_t format = 0, channels = 0, bits = 0;
  uint32_t rate = 0;
  const uint8_t *data = nullptr;
  size_t dataLen = 0;
  size_t pos = 12;
  while (pos + 8 <= len) {
    uint32_t chunkSize = 0;
    memcpy(&chunkSize, wav + pos + 4, 4);
    size_t chunkData = pos + 8;
    size_t available = len - chunkData;
    size_t actual = min((size_t)chunkSize, available);

    if (memcmp(wav + pos, "fmt ", 4) == 0 && actual >= 16) {
      memcpy(&format, wav + chunkData, 2);
      memcpy(&channels, wav + chunkData + 2, 2);
      memcpy(&rate, wav + chunkData + 4, 4);
      memcpy(&bits, wav + chunkData + 14, 2);
    } else if (memcmp(wav + pos, "data", 4) == 0 && actual > 0) {
      data = wav + chunkData;
      dataLen = actual;
      break;
    }

    if ((size_t)chunkSize > available) break;
    pos = chunkData + (size_t)chunkSize + (chunkSize & 1U);
  }

  if (!data || dataLen == 0) { err = "no wav data"; return false; }
  if (format != 1 || bits != 16 || (channels != 1 && channels != 2)) {
    err = "wav format";
    return false;
  }
  if (rate < 8000 || rate > 96000) { err = "wav rate"; return false; }
  dataLen -= dataLen % ((size_t)channels * 2U);

  ampMute();
  i2s_config_t cfg = {};
  cfg.mode = (i2s_mode_t)(I2S_MODE_MASTER | I2S_MODE_TX);
  cfg.sample_rate = rate;
  cfg.bits_per_sample = I2S_BITS_PER_SAMPLE_16BIT;
  cfg.channel_format = channels == 1 ? I2S_CHANNEL_FMT_ONLY_LEFT : I2S_CHANNEL_FMT_RIGHT_LEFT;
  cfg.communication_format = I2S_COMM_FORMAT_STAND_I2S;
  cfg.intr_alloc_flags = ESP_INTR_FLAG_LEVEL1;
  cfg.dma_buf_count = 8; cfg.dma_buf_len = 256;
  cfg.tx_desc_auto_clear = true;
  if (i2s_driver_install(I2S_NUM_0, &cfg, 0, NULL) != ESP_OK) {
    err = "i2s install";
    return false;
  }
  i2s_pin_config_t p = {};
  p.bck_io_num = I2S_BCLK_PIN; p.ws_io_num = I2S_WS_PIN;
  p.data_out_num = I2S_AMP_DIN_PIN; p.data_in_num = I2S_PIN_NO_CHANGE;
  if (i2s_set_pin(I2S_NUM_0, &p) != ESP_OK) {
    i2s_driver_uninstall(I2S_NUM_0);
    ampMute();
    err = "i2s pin";
    return false;
  }
  i2s_zero_dma_buffer(I2S_NUM_0);
  ampEnable();
  size_t off = 0;
  bool ok = true;
  while (off < dataLen) {
    size_t wr = 0;
    if (i2s_write(I2S_NUM_0, data + off, dataLen - off, &wr, portMAX_DELAY) != ESP_OK || wr == 0) {
      ok = false;
      err = "i2s write";
      break;
    }
    off += wr;
  }
  if (ok) delay(100);  // 等最后一批 DMA 音频真正播放完再卸载驱动
  i2s_driver_uninstall(I2S_NUM_0);
  ampMute();
  return ok;
}

static void speakerSelfTest(Stream &out) {
  out.print("TEST:SPK_START\r\n");
  const uint32_t rate = 16000;
  const size_t sampleCount = rate * 2;
  const size_t wavLen = 44 + sampleCount * 2;
  uint8_t *wav = (uint8_t *)ps_malloc(wavLen);
  if (!wav) wav = (uint8_t *)malloc(wavLen);
  if (!wav) { out.print("ERR:SPK:mem\r\n"); return; }

  writeWavHeader(wav, sampleCount * 2, rate, 1, 16);
  static const int16_t tone[16] = {
    0, 2296, 4243, 5543, 6000, 5543, 4243, 2296,
    0, -2296, -4243, -5543, -6000, -5543, -4243, -2296
  };
  int16_t *pcm = (int16_t *)(wav + 44);
  for (size_t i = 0; i < sampleCount; i++) pcm[i] = tone[i & 15U];

  String err;
  bool ok = playWav(wav, wavLen, err);
  free(wav);
  if (ok) out.print("TEST:SPK_OK\r\n");
  else { out.print("ERR:SPK:"); out.print(err); out.print("\r\n"); }
}

static void microphoneSelfTest(Stream &out) {
  out.print("TEST:MIC_START\r\n");
  if (!i2sRxInstall()) { out.print("ERR:MIC:i2s install\r\n"); return; }

  static uint32_t raw[128];
  const size_t target = (size_t)SAMPLE_RATE * 2;
  size_t got = 0;
  uint64_t totalAbs = 0;
  uint32_t peak = 0;
  uint32_t lastData = millis();
  while (got < target) {
    size_t br = 0;
    i2s_read(I2S_NUM_0, raw, sizeof(raw), &br, 100);
    size_t n = br / 4;
    if (n) lastData = millis();
    for (size_t i = 0; i < n && got < target; i++, got++) {
      int32_t sample = (int16_t)(raw[i] >> 14);
      uint32_t level = sample < 0 ? (uint32_t)(-sample) : (uint32_t)sample;
      totalAbs += level;
      if (level > peak) peak = level;
    }
    if (millis() - lastData > 3000) break;
    yield();
  }
  i2s_driver_uninstall(I2S_NUM_0);
  ampMute();

  if (got == 0) { out.print("ERR:MIC:no data\r\n"); return; }
  out.print("TEST:MIC,AVG="); out.print((uint32_t)(totalAbs / got));
  out.print(",PEAK="); out.print(peak); out.print("\r\n");
}

// ------------------- WiFi -------------------
static void connectWiFi() {
  if (WiFi.status() == WL_CONNECTED) return;
  DBG_PRINTF("[WiFi] connecting %s ...\r\n", WIFI_SSID);
  WiFi.mode(WIFI_STA);
  WiFi.begin(WIFI_SSID, WIFI_PASS);
  uint32_t t0 = millis();
  while (WiFi.status() != WL_CONNECTED && millis() - t0 < 20000) { delay(500); DBG_PRINT("."); }
  DBG_PRINTF("\r\n[WiFi] %s IP=%s\r\n", WiFi.status() == WL_CONNECTED ? "OK" : "FAIL",
             WiFi.localIP().toString().c_str());
}

// ------------------- 拼 content: 图片? + 音频? + 文本 -------------------
static char *buildBody(const uint8_t *img, size_t imgLen,
                       const uint8_t *wav, size_t wavLen, size_t *outLen) {
  String prompt = ASK_PROMPT;
  if (g_sensorData.length()) prompt = "设备数据: " + g_sensorData + "。" + prompt;
  String promptEsc = jsonEscape(prompt);

  const char *A = "{\"model\":\"" OMNI_MODEL "\",\"modalities\":[\"text\"],\"messages\":[{\"role\":\"user\",\"content\":[";
  const char *B = "{\"type\":\"image_url\",\"image_url\":{\"url\":\"data:image/jpeg;base64,";
  const char *C = "\"}},";
  const char *D = "{\"type\":\"input_audio\",\"input_audio\":{\"data\":\"data:audio/wav;base64,";
  const char *E = "\",\"format\":\"wav\"}},";
  String F = String("{\"type\":\"text\",\"text\":\"") + promptEsc + "\"}]}]}";

  size_t img64 = img ? ((imgLen + 2) / 3) * 4 : 0;
  size_t aud64 = wav ? ((wavLen + 2) / 3) * 4 : 0;
  size_t total = strlen(A) + strlen(B) + img64 + strlen(C) + strlen(D) + aud64 + strlen(E) + F.length();

  char *body = (char *)ps_malloc(total + 1);
  if (!body) body = (char *)malloc(total + 1);
  if (!body) return nullptr;

  size_t o = 0;
  memcpy(body + o, A, strlen(A)); o += strlen(A);
  if (img) { memcpy(body + o, B, strlen(B)); o += strlen(B); o += base64Encode(img, imgLen, body + o); memcpy(body + o, C, strlen(C)); o += strlen(C); }
  if (wav) { memcpy(body + o, D, strlen(D)); o += strlen(D); o += base64Encode(wav, wavLen, body + o); memcpy(body + o, E, strlen(E)); o += strlen(E); }
  memcpy(body + o, F.c_str(), F.length()); o += F.length();
  body[o] = 0;
  *outLen = o;
  return body;
}

static bool postChat(char *body, size_t len, String &answer, String &err) {
  WiFiClientSecure cli; cli.setInsecure();
  HTTPClient https; https.setTimeout(40000);
  if (!https.begin(cli, String("https://") + API_HOST + OMNI_PATH)) { err = "begin"; return false; }
  https.addHeader("Content-Type", "application/json");
  https.addHeader("Authorization", String("Bearer ") + API_KEY);
  int code = https.POST((uint8_t *)body, len);
  if (code <= 0) { https.end(); err = "send " + String(code); return false; }
  String resp = https.getString();
  https.end();
  DBG_PRINTF("[AI] code=%d\r\n", code);
  if (code != 200) { err = "http " + String(code) + " " + resp.substring(0, 160); return false; }
  StaticJsonDocument<256> filter;
  filter["choices"][0]["message"]["content"] = true;
  DynamicJsonDocument doc(16384);
  if (deserializeJson(doc, resp, DeserializationOption::Filter(filter))) { err = "json"; return false; }
  answer = doc["choices"][0]["message"]["content"].as<String>();
  if (!answer.length()) { err = "empty"; return false; }
  return true;
}

// 拍照(可传 nullptr 表示只录音) + 录音 -> 回答
static bool askAI(Stream &out, const uint8_t *img, size_t imgLen, String &answer, String &err) {
  size_t pcmLen = 0;
  bool speechDetected = false;
  uint8_t *pcm = recordPcm(out, &pcmLen, &speechDetected);
  if (!pcm) { err = "rec"; return false; }
  if (!speechDetected) { free(pcm); err = "no speech"; return false; }
  size_t wavLen = 44 + pcmLen;
  uint8_t *wav = (uint8_t *)ps_malloc(wavLen);
  if (!wav) wav = (uint8_t *)malloc(wavLen);
  if (!wav) { free(pcm); err = "mem"; return false; }
  writeWavHeader(wav, pcmLen, SAMPLE_RATE, 1, 16);
  memcpy(wav + 44, pcm, pcmLen);
  free(pcm);

  size_t bodyLen = 0;
  char *body = buildBody(img, imgLen, wav, wavLen, &bodyLen);
  free(wav);
  if (!body) { err = "body"; return false; }
  DBG_PRINTF("[AI] req %u bytes\r\n", (unsigned)bodyLen);
  bool ok = postChat(body, bodyLen, answer, err);
  free(body);
  return ok;
}

// ------------------- TTS 播放 -------------------
static bool speak(const String &text, String &err) {
  String body = String("{\"model\":\"") + TTS_MODEL + "\",\"input\":{\"text\":\"" +
                jsonEscape(text) + "\",\"voice\":\"" + TTS_VOICE +
                "\"},\"parameters\":{\"format\":\"wav\",\"sample_rate\":" + String(TTS_RATE) + "}}";
  WiFiClientSecure cli; cli.setInsecure();
  HTTPClient https; https.setTimeout(30000);
  if (!https.begin(cli, String("https://") + API_HOST + TTS_PATH)) { err = "tts begin"; return false; }
  https.addHeader("Content-Type", "application/json");
  https.addHeader("Authorization", String("Bearer ") + API_KEY);
  int code = https.POST(body);
  if (code != 200) { https.end(); err = "tts http " + String(code); return false; }
  String resp = https.getString();
  https.end();
  String url;
  DynamicJsonDocument doc(8192);
  if (!deserializeJson(doc, resp)) url = doc["output"]["audio"]["url"].as<String>();
  if (!url.length()) { err = "no tts url"; return false; }
  DBG_PRINTLN("[TTS] url=" + url);

  WiFiClientSecure c2s; c2s.setInsecure();
  WiFiClient c2p;
  HTTPClient h2; h2.setTimeout(30000);
  bool begun = url.startsWith("https") ? h2.begin(c2s, url) : h2.begin(c2p, url);
  if (!begun) { err = "dl begin"; return false; }
  int c = h2.GET();
  if (c != 200) { h2.end(); err = "dl " + String(c); return false; }
  int n = h2.getSize(); size_t cap = n > 0 ? n : 512000;
  uint8_t *buf = (uint8_t *)ps_malloc(cap);
  if (!buf) buf = (uint8_t *)malloc(cap);
  if (!buf) { h2.end(); err = "mem audio"; return false; }
  WiFiClient *st = h2.getStreamPtr();
  size_t tot = 0; uint32_t t0 = millis();
  while ((h2.connected() || st->available()) && tot < cap && millis() - t0 < 15000) {
    size_t av = st->available();
    if (av) { int r = st->readBytes(buf + tot, min(av, cap - tot)); tot += r; }
    else delay(5);
  }
  h2.end();
  DBG_PRINTF("[TTS] got %u bytes\r\n", (unsigned)tot);
  if (tot == 0) { free(buf); err = "empty audio"; return false; }
  String playErr;
  bool played = playWav(buf, tot, playErr);
  free(buf);
  if (!played) { err = "play " + playErr; return false; }
  return true;
}

// ------------------- 命令处理 -------------------
static void reply(Stream &out, bool ok, const String &s) {
  out.print(ok ? "OK:" : "ERR:");
  out.print(s);
  out.print("\r\n");
}

static void handleLine(String line, Stream &out) {
  line.trim();
  if (line.length() == 0) return;
  DBG_PRINT("[CMD] "); DBG_PRINTLN(line);

  String up = line; up.toUpperCase();

  if (up == "SPK") {
    speakerSelfTest(out);
    return;
  }

  if (up == "MIC") {
    microphoneSelfTest(out);
    return;
  }

  if (up.startsWith("PZ")) {
    out.print("BUSY:PZ\r\n");
    if (WiFi.status() != WL_CONNECTED) connectWiFi();
    if (WiFi.status() != WL_CONNECTED) { reply(out, false, "wifi"); return; }
    size_t imgLen = 0;
    uint8_t *img = captureJpeg(&imgLen);
    if (!img) { reply(out, false, "camera"); return; }
    out.print("REC...\r\n");
    String ans, err;
    bool ok = askAI(out, img, imgLen, ans, err);
    free(img);
    if (!ok) { reply(out, false, err); return; }
    ans.replace("\r", " "); ans.replace("\n", " ");
    reply(out, true, ans);
#if ENABLE_TTS
    out.print("TTS:START\r\n");
    String te;
    if (!speak(ans, te)) { out.print("ERR:TTS:"); out.print(te); out.print("\r\n"); }
    else out.print("TTS:OK\r\n");
#endif
    return;
  }

  if (up.startsWith("LY")) {
    out.print("BUSY:LY\r\n");
    if (WiFi.status() != WL_CONNECTED) connectWiFi();
    if (WiFi.status() != WL_CONNECTED) { reply(out, false, "wifi"); return; }
    out.print("REC...\r\n");
    String ans, err;
    bool ok = askAI(out, nullptr, 0, ans, err);
    if (!ok) { reply(out, false, err); return; }
    ans.replace("\r", " "); ans.replace("\n", " ");
    reply(out, true, ans);
#if ENABLE_TTS
    out.print("TTS:START\r\n");
    String te;
    if (!speak(ans, te)) { out.print("ERR:TTS:"); out.print(te); out.print("\r\n"); }
    else out.print("TTS:OK\r\n");
#endif
    return;
  }

  g_sensorData = line;
  DBG_PRINTLN("[DATA] " + g_sensorData);
}

static void pollStream(Stream &s, String &buf) {
  while (s.available()) {
    char c = (char)s.read();
    if (c == '\n' || c == '\r') {
      if (buf.length() > 0) { String l = buf; buf = ""; handleLine(l, s); }
    } else if (buf.length() < 300) buf += c;
  }
}

// ------------------- setup / loop -------------------
void setup() {
  Serial.begin(MCU_BAUD);
  Serial.setDebugOutput(false);
  delay(200);
  Serial.printf("SYS:RESET,%d\r\n", (int)esp_reset_reason());
  pinMode(LED_GPIO_NUM, OUTPUT);
  digitalWrite(LED_GPIO_NUM, FLASH_OFF);
  ampMute();
  if (!psramFound()) Serial.print("WARN:NO_PSRAM\r\n");
  connectWiFi();
  Serial.print(WiFi.status() == WL_CONNECTED ? "READY:WIFI_OK\r\n" : "READY:WIFI_FAIL\r\n");
}

void loop() {
  pollStream(Serial, rxLine);
  if (WiFi.status() != WL_CONNECTED) {
    static uint32_t last = 0;
    if (millis() - last > 15000) { last = millis(); connectWiFi(); }
  }
  delay(2);
}

使用前必须修改的配置

打开CameraAI.ino，找到文件顶部的"用户配置区"。

1. Wi-Fi名称和密码

复制代码 
#define WIFI_SSID "你的2.4G WiFi名称"
#define WIFI_PASS "你的WiFi密码"

ESP32不支持连接只开放5GHz的Wi-Fi，路由器必须启用2.4GHz网络。

2. API Key

复制代码 
#define API_KEY "替换为你自己的API Key"

千问平台API的获取（注意新增完毕后智慧显示一次APIkey记得保存）

3. API地址和模型（我使用的千问最偏见TTS和多模态模型 需要的话自己注册或者换）

当前代码中的位置：

复制代码 
#define API_HOST   "你的API服务域名"
#define OMNI_MODEL "你的多模态模型名称"
#define OMNI_PATH  "/多模态接口路径"
​
#define TTS_MODEL  "你的TTS模型名称"
#define TTS_PATH   "/TTS接口路径"

如果更换服务商，除了域名和模型名称外，还需要确认请求JSON、鉴权头、响应JSON和TTS音频URL字段是否兼容。只修改模型名称并不能保证接口兼容。

4. AI回答要求

复制代码 
#define ASK_PROMPT \
  "请结合提供的音频(和图片)以及设备数据, 用一句简体中文回答。"

这里可以决定AI如何回答。例如：

复制代码 
#define ASK_PROMPT \
  "请结合图片、语音和设备数据，用简短中文回答；如果设备数据中有温度，请优先引用实际温度。"

六、可调宏定义说明

1. TTS设置

复制代码 
#define ENABLE_TTS  1
#define TTS_VOICE   "Cherry"
#define TTS_RATE    16000
宏 作用 调整建议
ENABLE_TTS 是否播放AI语音 1开启，0只返回文字
TTS_VOICE TTS音色 必须是服务端支持的音色名称
TTS_RATE TTS输出采样率 建议与音频系统保持16000Hz

2. 录音和图像设置

复制代码 
#define SAMPLE_RATE     16000
#define RECORD_SECONDS  10
#define FRAME_SIZE      FRAMESIZE_VGA
#define JPEG_QUALITY    12
宏 作用 调整建议
SAMPLE_RATE 麦克风采样率 语音识别推荐16000Hz
RECORD_SECONDS 最长录音时间 增大会增加PSRAM和网络数据量
FRAME_SIZE 摄像头分辨率 VGA兼顾清晰度和请求大小
JPEG_QUALITY JPEG压缩质量 数字越小质量越高、文件越大

如果网络较慢或请求过大，可以把分辨率改为：

复制代码 
#define FRAME_SIZE FRAMESIZE_QVGA

3. 静音检测设置

复制代码 
#define VAD_CALIBRATE_MS      400
#define VAD_START_TIMEOUT_MS  4000
#define VAD_START_CONFIRM_MS  80
#define VAD_SILENCE_MS        1200
#define VAD_MIN_LEVEL         120
宏 作用 调大后的效果
VAD_CALIBRATE_MS 环境底噪估算时间 启动更慢，但底噪估算更充分
VAD_START_TIMEOUT_MS 最长等待开口时间 给用户更多准备时间
VAD_START_CONFIRM_MS 连续声音确认时间 不容易被瞬时噪声误触发
VAD_SILENCE_MS 说完后的静音结束时间 更不容易截断句间停顿
VAD_MIN_LEVEL 最低声音门限 更不容易被环境噪声触发

常见调整：

  • 环境噪声导致录音一直不停：将VAD_MIN_LEVEL提高到200～300。

  • 正常说话检测不到：将VAD_MIN_LEVEL降低到60～100。

  • 句子还没说完就停止：将VAD_SILENCE_MS改为1600～2000。

  • 不说话等待时间太长：将VAD_START_TIMEOUT_MS改为2500～3000。

建议先发送MIC，根据返回的AVG和PEAK观察实际声音幅度，再调整门限。

4. 引脚定义

复制代码 
#define I2S_BCLK_PIN     14
#define I2S_WS_PIN       15
#define I2S_MIC_SD_PIN   2
#define I2S_AMP_DIN_PIN  12
#define I2S_AMP_SD_PIN   13

这些宏必须与实际接线一致。AI-Thinker ESP32-CAM的GPIO16连接板载PSRAM，不能改作BCLK或普通外设。GPIO4连接板载闪光灯，也不应复用。

5. 串口设置和调试输出

复制代码 
#define MCU_BAUD   115200
#define UART_DEBUG 0

  • MCU_BAUD必须与单片机UART波特率一致。

  • UART_DEBUG=0时，UART0只输出结构化业务信息。

  • UART_DEBUG=1会输出Wi-Fi、AI和TTS调试日志，但会与单片机协议混合，只适合串口助手调试。

相关推荐
BSD_CGQ1 小时前
人形机器人触觉传感器为什么集中选 FSR 柔性薄膜方案？选型边界与国产替代现状
人工智能·机器人·压力传感器·柔性薄膜压力传感器
Phoenixtree_DongZhao1 小时前
[CVPR2026] 视觉 Mamba：RNN 与线性 Transformer 的统一视角
人工智能·rnn·transformer
碧海银沙音频科技研究院1 小时前
基于炬芯ATS3089X的ST LIS2DW12 三轴 MEMS 加速度传感器（G-Sensor)敲击按键功能实现
人工智能·嵌入式硬件
ccbw161 小时前
CH32L103低功耗RISC-V MCU的集成外设与选型参考
单片机·嵌入式硬件·risc-v·低功耗mcu·潜创微·青稞risc-v4c
吴佳浩 Alben1 小时前
一文讲透推理引擎性能指标：TTFT、TPOT、KV命中率、并发，到底对应 vLLM / SGLang 的哪个参数？
人工智能·python·语言模型·架构·ai编程·vllm·sglang
ting94520001 小时前
Kilo Code for JetBrains 深度技术剖析：原生开源编程智能体内部实现与工程实践
人工智能·架构·开源
武子康1 小时前
PUBACK 已到，灯还没亮：Agent 设备指令何时算完成？
人工智能·后端·agent
逐光者9331 小时前
电子设计——运算放大器（运放）基础
嵌入式硬件
陈嘿萌1 小时前
CVPR 2026 | 全新的五模态信息融合数据集，包括RGB、事件、热成像、4D 雷达与 LiDAR
人工智能·深度学习·计算机视觉·自动驾驶·学术研究与理论基础·点云雷达
热门推荐
01国内可直接用、免费额度/永久免费的大模型API清单（含 SiliconFlow、火山、阿里、智谱、百度、Kimi、DeepSeek、DMXAPI 等）02历年考研数学一、数学二、数学三真题试卷及答案PDF03GitHub 镜像站点04「全语录+省流版」邵艾伦 × 孙宇晨：聊风口、聊个人IP、聊年轻人如何抓住 AI 时代的机会05蜘蛛磁力 搜索引擎大全，如何使用蜘蛛磁力查找磁力链接06如何新建文件夹？ 电脑新建文件夹的4种方法07Windows 10/11 如何退出微软账户并改用本地账户登录（含找不到选项的解决方法）08Telegram APK 下载安装完整指南 — 2026年最新09几个好用的ip纯净度检测网站10轻量化AI渗透武器库｜ARTEX自主攻防系统、资产梳理、漏洞挖掘、链路分析全流程智能化落地