【ZeroRange WebRTC】视频文件RTP打包与发送技术深度分析

视频文件RTP打包与发送技术深度分析

概述

视频文件RTP打包是将存储的视频文件（如H.264、H.265、VP8编码的视频）分解为适合网络传输的RTP数据包的过程。这个过程涉及视频编解码器的特定格式解析、NAL单元分割、RTP负载封装、分片策略以及网络适配等多个技术环节。WebRTC通过精密的打包机制，确保视频数据能够高效、可靠地通过IP网络传输。

基本原理

1. 视频文件打包流程

视频文件RTP打包的基本流程如下：

视频文件 → 读取帧数据 → 解析编码格式 → NAL单元分割 → RTP负载封装 → 网络发送
     ↓           ↓            ↓            ↓           ↓          ↓
   H.264文件   按帧读取   识别NAL边界   分片策略   添加RTP头   UDP传输
   H.265文件   提取数据   解析编码参数   MTU适配   设置时间戳   拥塞控制
   VP8文件     缓存管理   关键帧检测   序号管理   负载格式   错误恢复

2. 关键技术挑战

编解码器差异：

• H.264/H.265使用NAL单元结构
• VP8使用基于分片的打包方式
• 不同的负载格式和分片策略

网络适配：

• MTU大小限制（通常1200-1400字节）
• 实时性要求（低延迟）
• 网络抖动和丢包处理

性能优化：

• 内存使用效率
• CPU计算开销
• 并发处理能力

H.264视频文件RTP打包

1. H.264编码基础

H.264使用NAL（Network Abstraction Layer）单元结构：

NAL单元格式：
+---------------+---------------+---------------+---------------+
|0|1|2|3|4|5|6|7|0|1|2|3|4|5|6|7|0|1|2|3|4|5|6|7|0|1|2|3|4|5|6|7|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|F|NRI|  Type   |                                               |
+-+-+-+-+-------+                                               |
|                                                               |
|               NAL单元数据（可变长度）                         |
|                                                               |
|                               +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                               :...OPTIONAL RBSP trailing bits |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

NAL单元类型：

• 1-23: VCL（Video Coding Layer）单元，包含视频数据
• 5: IDR帧（关键帧）
• 1: 非IDR帧
• 24-31: 非VCL单元，包含参数集等信息

2. NAL单元解析

基于Amazon Kinesis WebRTC SDK的H.264解析实现：

// NAL单元边界检测
STATUS getNextNaluLength(PBYTE nalus, UINT32 nalusLength, PUINT32 pStart, PUINT32 pNaluLength) {
    UINT32 zeroCount = 0, offset = 0;
    BOOL naluFound = FALSE;
    PBYTE pCurrent = NULL;

    // Annex-B格式使用0x00000001或0x000001作为起始码
    while (offset < 4 && offset < nalusLength && nalus[offset] == 0) {
        offset++;
    }

    CHK(offset < nalusLength && offset < 4 && offset >= 2 && nalus[offset] == 1, 
        STATUS_RTP_INVALID_NALU);
    
    *pStart = ++offset;
    pCurrent = nalus + offset;

    // 查找下一个NAL单元起始码
    while (offset < nalusLength) {
        if (*pCurrent == 0) {
            offset++;
            pCurrent++;
        } else if (*pCurrent == 1) {
            if (*(pCurrent - 1) == 0 && *(pCurrent - 2) == 0) {
                zeroCount = *(pCurrent - 3) == 0 ? 3 : 2;
                naluFound = TRUE;
                break;
            }
            offset += 3;
            pCurrent += 3;
        } else {
            offset += 3;
            pCurrent += 3;
        }
    }
    
    *pNaluLength = MIN(offset, nalusLength) - *pStart - (naluFound ? zeroCount : 0);
    return STATUS_SUCCESS;
}

3. RTP打包策略

H.264支持三种RTP打包模式：

3.1 单NAL单元模式（Single NAL Unit）

适用于较小的NAL单元（小于MTU）：

// 单NAL单元打包
STATUS createSingleNaluPayload(PBYTE nalu, UINT32 naluLength, PRtpPacket pRtpPacket) {
    // 直接复制NAL单元数据
    MEMCPY(pRtpPacket->payload, nalu, naluLength);
    pRtpPacket->payloadLength = naluLength;
    
    // 设置RTP头
    pRtpPacket->header.markerBit = 1;  // 最后一个包
    pRtpPacket->header.sequenceNumber++;
    
    return STATUS_SUCCESS;
}

3.2 分片单元模式（FU-A）

适用于较大的NAL单元（超过MTU）：

// FU-A分片打包
STATUS createFragmentedUnitPayload(UINT32 mtu, PBYTE nalu, UINT32 naluLength, 
                                 PRtpPacket* pRtpPackets, PUINT32 packetCount) {
    UINT8 naluType = *nalu & 0x1F;      // NAL类型
    UINT8 naluRefIdc = *nalu & 0x60;    // NRI（重要性）
    UINT32 maxPayloadSize = mtu - FU_A_HEADER_SIZE;
    UINT32 remainingLength = naluLength - 1;  // 跳过NAL头
    PBYTE pCurPtr = nalu + 1;
    UINT32 fragmentCount = 0;
    
    while (remainingLength > 0) {
        UINT32 currentSize = MIN(maxPayloadSize, remainingLength);
        PRtpPacket pPacket = &pRtpPackets[fragmentCount];
        
        // FU-A指示器
        pPacket->payload[0] = 28 | naluRefIdc;  // FU-A类型 = 28
        
        // FU-A头
        pPacket->payload[1] = naluType;
        if (fragmentCount == 0) {
            // 起始分片
            pPacket->payload[1] |= 1 << 7;  // S位 = 1
        } else if (remainingLength == currentSize) {
            // 结束分片
            pPacket->payload[1] |= 1 << 6;  // E位 = 1
        }
        
        // 复制分片数据
        MEMCPY(pPacket->payload + FU_A_HEADER_SIZE, pCurPtr, currentSize);
        pPacket->payloadLength = FU_A_HEADER_SIZE + currentSize;
        
        // 设置RTP头
        pPacket->header.markerBit = (remainingLength == currentSize) ? 1 : 0;
        pPacket->header.sequenceNumber++;
        
        pCurPtr += currentSize;
        remainingLength -= currentSize;
        fragmentCount++;
    }
    
    *packetCount = fragmentCount;
    return STATUS_SUCCESS;
}

3.3 聚合包模式（STAP-A）

适用于多个小NAL单元的组合：

// STAP-A聚合打包（简化示例）
STATUS createStapAPayload(PBYTE* nalus, UINT32* naluLengths, UINT32 naluCount,
                        UINT32 mtu, PRtpPacket pRtpPacket) {
    UINT32 totalSize = 1;  // STAP-A头
    UINT32 offset = 1;
    
    // 计算总大小
    for (UINT32 i = 0; i < naluCount; i++) {
        totalSize += 2 + naluLengths[i];  // NALU大小 + NALU数据
    }
    
    CHK(totalSize <= mtu, STATUS_RTP_PAYLOAD_TOO_LARGE);
    
    // STAP-A头
    pRtpPacket->payload[0] = 24;  // STAP-A类型 = 24
    
    // 聚合NAL单元
    for (UINT32 i = 0; i < naluCount; i++) {
        // NALU大小（16位）
        pRtpPacket->payload[offset] = (naluLengths[i] >> 8) & 0xFF;
        pRtpPacket->payload[offset + 1] = naluLengths[i] & 0xFF;
        offset += 2;
        
        // NALU数据
        MEMCPY(pRtpPacket->payload + offset, nalus[i], naluLengths[i]);
        offset += naluLengths[i];
    }
    
    pRtpPacket->payloadLength = offset;
    pRtpPacket->header.markerBit = 1;
    
    return STATUS_SUCCESS;
}

4. 关键帧检测与处理

// 关键帧检测
BOOL isH264KeyFrame(PBYTE nalu, UINT32 naluLength) {
    if (naluLength < 1) return FALSE;
    
    UINT8 naluType = nalu[0] & 0x1F;
    
    // IDR帧（类型5）是关键帧
    if (naluType == 5) {
        return TRUE;
    }
    
    // SPS（类型7）和PPS（类型8）通常与关键帧一起发送
    if (naluType == 7 || naluType == 8) {
        return TRUE;
    }
    
    return FALSE;
}

// 关键帧优先处理
STATUS processKeyFramePriority(PH264Frame pFrame, PRtpTransceiver pTransceiver) {
    if (isH264KeyFrame(pFrame->naluData, pFrame->naluLength)) {
        // 关键帧需要特殊处理
        DLOGI("Processing key frame, type: %u", pFrame->naluData[0] & 0x1F);
        
        // 确保SPS/PPS在IDR帧之前发送
        if ((pFrame->naluData[0] & 0x1F) == 5) {
            CHK_STATUS(sendParameterSetsIfNeeded(pTransceiver));
        }
        
        // 标记关键帧时间戳
        pTransceiver->lastKeyFrameTimestamp = pFrame->timestamp;
    }
    
    return STATUS_SUCCESS;
}

H.265视频文件RTP打包

1. H.265与H.264的差异

H.265（HEVC）在RTP打包上与H.264的主要区别：

// H.265 NAL单元头解析
typedef struct {
    UINT8 forbiddenZeroBit;
    UINT8 nalUnitType;      // 6位，范围0-63
    UINT8 nuhLayerId;       // 6位
    UINT8 nuhTemporalId;    // 3位
} H265NalUnitHeader;

STATUS parseH265NaluHeader(PBYTE nalu, UINT32 naluLength, PH265NalUnitHeader pHeader) {
    if (naluLength < 2) return STATUS_RTP_INVALID_NALU;
    
    // H.265使用2字节NAL头
    UINT16 naluHeader = (nalu[0] << 8) | nalu[1];
    
    pHeader->forbiddenZeroBit = (naluHeader >> 15) & 0x01;
    pHeader->nalUnitType = (naluHeader >> 9) & 0x3F;  // 6位
    pHeader->nuhLayerId = (naluHeader >> 3) & 0x3F;  // 6位
    pHeader->nuhTemporalId = naluHeader & 0x07;      // 3位
    
    CHK(pHeader->forbiddenZeroBit == 0, STATUS_RTP_INVALID_NALU);
    
    return STATUS_SUCCESS;
}

2. H.265 RTP打包模式

H.265支持类似的打包模式，但类型编码不同：

// H.265 RTP打包类型定义
#define H265_PKTTYPE_SINGLE_NALU    0  // 单NAL单元
#define H265_PKTTYPE_FRAGMENTATION  1  // 分片单元（FU）
#define H265_PKTTYPE_AGGREGATION    2  // 聚合包（AP）

STATUS createPayloadForH265(UINT32 mtu, PBYTE nalus, UINT32 nalusLength, 
                           PBYTE payloadBuffer, PUINT32 pPayloadLength, 
                           PUINT32 pPayloadSubLength, PUINT32 pPayloadSubLenSize) {
    // H.265打包逻辑与H.264类似，但NAL类型不同
    UINT16 naluHeader = (nalus[0] << 8) | nalus[1];
    UINT8 nalUnitType = (naluHeader >> 9) & 0x3F;
    
    // 根据NAL单元类型选择打包策略
    switch (nalUnitType) {
        case 19: // IDR_W_RADL
        case 20: // IDR_N_LP
            // 关键帧处理
            return createH265KeyFramePayload(mtu, nalus, nalusLength, payloadBuffer, 
                                           pPayloadLength, pPayloadSubLength, pPayloadSubLenSize);
            
        case 1:  // TRAIL_N
        case 0:  // TRAIL_R
            // 普通帧处理
            return createH265NormalFramePayload(mtu, nalus, nalusLength, payloadBuffer, 
                                              pPayloadLength, pPayloadSubLength, pPayloadSubLenSize);
            
        case 32: // VPS_NUT
        case 33: // SPS_NUT
        case 34: // PPS_NUT
            // 参数集处理
            return createH265ParameterSetPayload(mtu, nalus, nalusLength, payloadBuffer, 
                                               pPayloadLength, pPayloadSubLength, pPayloadSubLenSize);
            
        default:
            return createH265DefaultPayload(mtu, nalus, nalusLength, payloadBuffer, 
                                        pPayloadLength, pPayloadSubLength, pPayloadSubLenSize);
    }
}

3. H.265分片单元（FU）

H.265的分片单元格式：

// H.265分片单元头
typedef struct {
    UINT8 fuHeader;     // FU头
    UINT8 donlField[2]; // Decoding Order Number (可选)
} H265FuHeader;

STATUS createH265FragmentationUnit(UINT32 mtu, PBYTE nalu, UINT32 naluLength, 
                                 PH265RtpPacket pPackets, PUINT32 packetCount) {
    UINT16 naluHeader = (nalu[0] << 8) | nalu[1];
    UINT8 nalUnitType = (naluHeader >> 9) & 0x3F;
    UINT8 nuhLayerId = (naluHeader >> 3) & 0x3F;
    UINT8 nuhTemporalId = naluHeader & 0x07;
    
    UINT32 maxPayloadSize = mtu - H265_FU_HEADER_SIZE;
    UINT32 remainingLength = naluLength - 2;  // 跳过2字节NAL头
    PBYTE pCurPtr = nalu + 2;
    UINT32 fragmentCount = 0;
    
    while (remainingLength > 0) {
        UINT32 currentSize = MIN(maxPayloadSize, remainingLength);
        PH265RtpPacket pPacket = &pPackets[fragmentCount];
        
        // 构建H.265 FU头
        // FU头结构：S(1) | E(1) | FuType(6)
        pPacket->payload[0] = 0;
        if (fragmentCount == 0) {
            pPacket->payload[0] |= 0x80;  // S位 = 1
        }
        if (remainingLength == currentSize) {
            pPacket->payload[0] |= 0x40;  // E位 = 1
        }
        pPacket->payload[0] |= (nalUnitType & 0x3F);  // FU类型
        
        // 复制分片数据
        MEMCPY(pPacket->payload + H265_FU_HEADER_SIZE, pCurPtr, currentSize);
        pPacket->payloadLength = H265_FU_HEADER_SIZE + currentSize;
        
        // 设置RTP头
        pPacket->header.markerBit = (remainingLength == currentSize) ? 1 : 0;
        pPacket->header.sequenceNumber++;
        
        pCurPtr += currentSize;
        remainingLength -= currentSize;
        fragmentCount++;
    }
    
    *packetCount = fragmentCount;
    return STATUS_SUCCESS;
}

VP8视频文件RTP打包

1. VP8编码特点

VP8使用不同的打包策略，基于分片（Partitions）而非NAL单元：

// VP8负载描述符
typedef struct {
    UINT8 startOfPartition;     // 起始分片标识
    UINT8 partitionId;         // 分片ID（3位）
    BOOL hasPictureId;         // 是否有图片ID
    BOOL hasTl0PicIdx;         // 是否有TL0PICIDX
    BOOL hasTID;               // 是否有TID
    BOOL hasKeyIdx;            // 是否有KEYIDX
} Vp8PayloadDescriptor;

STATUS createPayloadForVP8(UINT32 mtu, PBYTE pData, UINT32 dataLen, 
                         PBYTE payloadBuffer, PUINT32 pPayloadLength, 
                         PUINT32 pPayloadSubLength, PUINT32 pPayloadSubLenSize) {
    UINT32 payloadRemaining = dataLen;
    UINT32 payloadLenConsumed = 0;
    PBYTE currentData = pData;
    BOOL sizeCalculationOnly = (payloadBuffer == NULL);
    PayloadArray payloadArray;
    
    MEMSET(&payloadArray, 0, SIZEOF(payloadArray));
    payloadArray.payloadBuffer = payloadBuffer;
    
    while (payloadRemaining > 0) {
        payloadLenConsumed = MIN(mtu - VP8_PAYLOAD_DESCRIPTOR_SIZE, payloadRemaining);
        payloadArray.payloadLength += (payloadLenConsumed + VP8_PAYLOAD_DESCRIPTOR_SIZE);
        
        if (!sizeCalculationOnly) {
            // VP8负载描述符
            *payloadArray.payloadBuffer = (payloadArray.payloadSubLenSize == 0) ? 
                VP8_PAYLOAD_DESCRIPTOR_START_OF_PARTITION_VALUE : 0;
            payloadArray.payloadBuffer++;
            
            // 复制VP8数据
            MEMCPY(payloadArray.payloadBuffer, currentData, payloadLenConsumed);
            
            // 记录子负载长度
            pPayloadSubLength[payloadArray.payloadSubLenSize] = 
                (payloadLenConsumed + VP8_PAYLOAD_DESCRIPTOR_SIZE);
            payloadArray.payloadBuffer += payloadLenConsumed;
            currentData += payloadLenConsumed;
        }
        
        payloadArray.payloadSubLenSize++;
        payloadRemaining -= payloadLenConsumed;
    }
    
    if (!sizeCalculationOnly) {
        *pPayloadLength = payloadArray.payloadLength;
        *pPayloadSubLenSize = payloadArray.payloadSubLenSize;
    }
    
    return STATUS_SUCCESS;
}

2. VP8关键帧检测

// VP8关键帧检测
BOOL isVP8KeyFrame(PBYTE vp8Data, UINT32 dataLen) {
    if (dataLen < 10) return FALSE;
    
    // VP8帧头解析
    UINT8 frameTag = vp8Data[0];
    
    // 检查关键帧标识
    if ((frameTag & 0x01) == 0) {
        // 关键帧（I帧）
        return TRUE;
    }
    
    return FALSE;
}

// VP8帧头解析
STATUS parseVP8FrameHeader(PBYTE vp8Data, UINT32 dataLen, PVp8FrameInfo pFrameInfo) {
    if (dataLen < 3) return STATUS_RTP_INVALID_VP8_DATA;
    
    UINT8 frameTag = vp8Data[0];
    
    // 解析帧类型
    pFrameInfo->isKeyFrame = (frameTag & 0x01) == 0;
    pFrameInfo->version = (frameTag >> 1) & 0x07;
    pFrameInfo->showFrame = (frameTag >> 4) & 0x01;
    
    // 如果是关键帧，解析更多头信息
    if (pFrameInfo->isKeyFrame && dataLen >= 10) {
        // 解析关键帧头
        pFrameInfo->width = (vp8Data[6] | (vp8Data[7] << 8)) & 0x3FFF;
        pFrameInfo->height = (vp8Data[8] | (vp8Data[9] << 8)) & 0x3FFF;
        pFrameInfo->horizontalScale = vp8Data[7] >> 6;
        pFrameInfo->verticalScale = vp8Data[9] >> 6;
    }
    
    return STATUS_SUCCESS;
}

视频文件读取与帧提取

1. 文件读取机制

基于Amazon Kinesis WebRTC SDK的视频文件读取：

// 从磁盘读取视频帧
STATUS readFrameFromDisk(PBYTE pFrame, PUINT32 pSize, PCHAR frameFilePath) {
    STATUS retStatus = STATUS_SUCCESS;
    UINT64 size = 0;
    
    CHK_ERR(pSize != NULL, STATUS_NULL_ARG, "[KVS Master] Invalid file size");
    size = *pSize;
    
    // 读取文件内容
    CHK_STATUS(readFile(frameFilePath, TRUE, pFrame, &size));

CleanUp:
    if (pSize != NULL) {
        *pSize = (UINT32) size;
    }
    
    return retStatus;
}

// 视频帧发送循环
PVOID sendVideoPackets(PVOID args) {
    PSampleConfiguration pSampleConfiguration = (PSampleConfiguration) args;
    Frame frame;
    UINT32 fileIndex = 0, frameSize;
    CHAR filePath[MAX_PATH_LEN + 1];
    
    frame.presentationTs = 0;
    
    while (!ATOMIC_LOAD_BOOL(&pSampleConfiguration->appTerminateFlag)) {
        // 循环读取视频帧文件
        fileIndex = fileIndex % NUMBER_OF_H264_FRAME_FILES + 1;
        
        if (pSampleConfiguration->videoCodec == RTC_CODEC_H264_PROFILE_42E01F_LEVEL_ASYMMETRY_ALLOWED_PACKETIZATION_MODE) {
            SNPRINTF(filePath, MAX_PATH_LEN, "./h264SampleFrames/frame-%04d.h264", fileIndex);
        } else if (pSampleConfiguration->videoCodec == RTC_CODEC_H265) {
            SNPRINTF(filePath, MAX_PATH_LEN, "./h265SampleFrames/frame-%04d.h265", fileIndex);
        }
        
        // 获取帧大小
        CHK_STATUS(readFrameFromDisk(NULL, &frameSize, filePath));
        
        // 动态分配帧缓冲区
        if (frameSize > pSampleConfiguration->videoBufferSize) {
            pSampleConfiguration->pVideoFrameBuffer = (PBYTE) MEMREALLOC(
                pSampleConfiguration->pVideoFrameBuffer, frameSize);
            pSampleConfiguration->videoBufferSize = frameSize;
        }
        
        // 读取帧数据
        frame.frameData = pSampleConfiguration->pVideoFrameBuffer;
        frame.size = frameSize;
        CHK_STATUS(readFrameFromDisk(frame.frameData, &frameSize, filePath));
        
        // 更新帧时间戳
        frame.presentationTs += SAMPLE_VIDEO_FRAME_DURATION;
        
        // 发送到所有活跃的流媒体会话
        MUTEX_LOCK(pSampleConfiguration->streamingSessionListReadLock);
        for (UINT32 i = 0; i < pSampleConfiguration->streamingSessionCount; ++i) {
            STATUS status = writeFrame(
                pSampleConfiguration->sampleStreamingSessionList[i]->pVideoRtcRtpTransceiver, 
                &frame);
            
            if (status != STATUS_SRTP_NOT_READY_YET && status != STATUS_SUCCESS) {
                DLOGV("writeFrame() failed with 0x%08x", status);
            } else if (status == STATUS_SRTP_NOT_READY_YET) {
                // SRTP未就绪，重置文件索引确保关键帧同步
                fileIndex = 0;
            }
        }
        MUTEX_UNLOCK(pSampleConfiguration->streamingSessionListReadLock);
        
        // 帧间延时控制
        THREAD_SLEEP(SAMPLE_VIDEO_FRAME_DURATION);
    }
    
    return NULL;
}

2. 实时视频源处理

对于实时视频源（如摄像头、RTSP流）：

// GStreamer视频源处理
typedef struct {
    GstElement* pipeline;
    GstElement* appsrc;
    GstElement* decoder;
    GstElement* converter;
    GstElement* encoder;
    GstElement* appsink;
    
    PBYTE frameBuffer;
    UINT32 frameBufferSize;
    MUTEX bufferLock;
    
    BOOL isRunning;
} GstVideoSource;

// GStreamer回调函数
static GstFlowReturn onNewSample(GstElement* sink, gpointer data) {
    GstVideoSource* pVideoSource = (GstVideoSource*)data;
    GstSample* sample;
    GstBuffer* buffer;
    GstMapInfo map;
    
    // 获取样本
    g_signal_emit_by_name(sink, "pull-sample", &sample);
    if (sample == NULL) {
        return GST_FLOW_ERROR;
    }
    
    buffer = gst_sample_get_buffer(sample);
    if (buffer == NULL) {
        gst_sample_unref(sample);
        return GST_FLOW_ERROR;
    }
    
    // 映射缓冲区
    if (gst_buffer_map(buffer, &map, GST_MAP_READ)) {
        MUTEX_LOCK(pVideoSource->bufferLock);
        
        // 动态调整缓冲区大小
        if (map.size > pVideoSource->frameBufferSize) {
            pVideoSource->frameBuffer = (PBYTE)MEMREALLOC(pVideoSource->frameBuffer, map.size);
            pVideoSource->frameBufferSize = map.size;
        }
        
        // 复制帧数据
        MEMCPY(pVideoSource->frameBuffer, map.data, map.size);
        
        MUTEX_UNLOCK(pVideoSource->bufferLock);
        
        gst_buffer_unmap(buffer, &map);
    }
    
    gst_sample_unref(sample);
    return GST_FLOW_OK;
}

// 创建GStreamer视频管道
STATUS createGstVideoPipeline(GstVideoSource* pVideoSource, RTC_CODEC codec) {
    GstElement* pipeline;
    GstCaps* caps;
    gchar* pipelineStr;
    
    switch (codec) {
        case RTC_CODEC_H264_PROFILE_42E01F_LEVEL_ASYMMETRY_ALLOWED_PACKETIZATION_MODE:
            pipelineStr = g_strdup_printf(
                "videotestsrc ! video/x-raw,width=640,height=480,framerate=30/1 ! "
                "x264enc bframes=0 speed-preset=veryfast bitrate=512 byte-stream=TRUE ! "
                "video/x-h264,stream-format=byte-stream,alignment=au,profile=baseline ! "
                "appsink name=appsink sync=TRUE emit-signals=TRUE");
            break;
            
        case RTC_CODEC_H265:
            pipelineStr = g_strdup_printf(
                "videotestsrc ! video/x-raw,width=640,height=480,framerate=30/1 ! "
                "x265enc speed-preset=veryfast bitrate=512 tune=zerolatency ! "
                "video/x-h265,stream-format=byte-stream,alignment=au,profile=main ! "
                "appsink name=appsink sync=TRUE emit-signals=TRUE");
            break;
            
        case RTC_CODEC_VP8:
            pipelineStr = g_strdup_printf(
                "videotestsrc ! video/x-raw,width=640,height=480,framerate=30/1 ! "
                "vp8enc deadline=1 ! "
                "video/x-vp8 ! "
                "appsink name=appsink sync=TRUE emit-signals=TRUE");
            break;
            
        default:
            return STATUS_NOT_IMPLEMENTED;
    }
    
    // 创建管道
    GError* error = NULL;
    pipeline = gst_parse_launch(pipelineStr, &error);
    
    if (error != NULL) {
        DLOGE("Failed to create GStreamer pipeline: %s", error->message);
        g_error_free(error);
        return STATUS_GSTREAMER_PIPELINE_ERROR;
    }
    
    // 获取appsink元素
    pVideoSource->appsink = gst_bin_get_by_name(GST_BIN(pipeline), "appsink");
    
    // 连接回调
    g_signal_connect(pVideoSource->appsink, "new-sample", G_CALLBACK(onNewSample), pVideoSource);
    
    // 设置管道状态
    gst_element_set_state(pipeline, GST_STATE_PLAYING);
    
    pVideoSource->pipeline = pipeline;
    pVideoSource->isRunning = TRUE;
    
    g_free(pipelineStr);
    return STATUS_SUCCESS;
}

性能优化策略

1. 内存管理优化

// 内存池管理
typedef struct {
    PBYTE* bufferPool;
    UINT32 poolSize;
    UINT32 bufferSize;
    UINT32 availableCount;
    MUTEX poolLock;
} FrameBufferPool;

FrameBufferPool* createFrameBufferPool(UINT32 poolSize, UINT32 bufferSize) {
    FrameBufferPool* pPool = (FrameBufferPool*)MEMALLOC(SIZEOF(FrameBufferPool));
    
    pPool->poolSize = poolSize;
    pPool->bufferSize = bufferSize;
    pPool->availableCount = poolSize;
    
    pPool->bufferPool = (PBYTE*)MEMALLOC(SIZEOF(PBYTE) * poolSize);
    
    for (UINT32 i = 0; i < poolSize; i++) {
        pPool->bufferPool[i] = (PBYTE)MEMALLOC(bufferSize);
    }
    
    MUTEX_INIT(pPool->poolLock);
    
    return pPool;
}

PBYTE acquireFrameBuffer(FrameBufferPool* pPool) {
    PBYTE buffer = NULL;
    
    MUTEX_LOCK(pPool->poolLock);
    
    if (pPool->availableCount > 0) {
        buffer = pPool->bufferPool[pPool->availableCount - 1];
        pPool->availableCount--;
    }
    
    MUTEX_UNLOCK(pPool->poolLock);
    
    return buffer;
}

VOID releaseFrameBuffer(FrameBufferPool* pPool, PBYTE buffer) {
    MUTEX_LOCK(pPool->poolLock);
    
    if (pPool->availableCount < pPool->poolSize) {
        pPool->bufferPool[pPool->availableCount] = buffer;
        pPool->availableCount++;
    }
    
    MUTEX_UNLOCK(pPool->poolLock);
}

2. 零拷贝优化

// 零拷贝RTP打包
typedef struct {
    PBYTE data;
    UINT32 size;
    BOOL isReference;  // 是否引用外部数据
    PVOID originalBuffer;
} ZeroCopyBuffer;

STATUS createZeroCopyRtpPacket(ZeroCopyBuffer* pBuffer, PRtpPacket pRtpPacket) {
    if (pBuffer->isReference) {
        // 使用引用，避免数据复制
        pRtpPacket->payload = pBuffer->data;
        pRtpPacket->payloadLength = pBuffer->size;
        pRtpPacket->isZeroCopy = TRUE;
    } else {
        // 需要数据复制
        MEMCPY(pRtpPacket->payload, pBuffer->data, pBuffer->size);
        pRtpPacket->payloadLength = pBuffer->size;
        pRtpPacket->isZeroCopy = FALSE;
    }
    
    return STATUS_SUCCESS;
}

3. 并行处理优化

// 并行RTP打包
typedef struct {
    ThreadPool* packThreadPool;
    Queue* pendingFrames;
    AtomicBool isRunning;
    MUTEX queueLock;
} ParallelRtpPacker;

// 并行打包工作函数
PVOID parallelPackWorker(PVOID args) {
    ParallelRtpPacker* pPacker = (ParallelRtpPacker*)args;
    Frame* pFrame;
    
    while (ATOMIC_LOAD_BOOL(&pPacker->isRunning)) {
        MUTEX_LOCK(pPacker->queueLock);
        
        if (queueIsEmpty(pPacker->pendingFrames)) {
            MUTEX_UNLOCK(pPacker->queueLock);
            THREAD_SLEEP(1);  // 短暂休眠
            continue;
        }
        
        queueDequeue(pPacker->pendingFrames, &pFrame);
        MUTEX_UNLOCK(pPacker->queueLock);
        
        // 并行执行RTP打包
        RtpPacket* pPackets = (RtpPacket*)MEMALLOC(SIZEOF(RtpPacket) * MAX_RTP_PACKETS_PER_FRAME);
        UINT32 packetCount = 0;
        
        createRtpPackets(pFrame, pPackets, &packetCount);
        
        // 发送打包好的数据包
        sendRtpPackets(pPackets, packetCount);
        
        // 释放资源
        MEMFREE(pPackets);
        MEMFREE(pFrame);
    }
    
    return NULL;
}

// 提交帧进行并行打包
STATUS submitFrameForParallelPacking(ParallelRtpPacker* pPacker, Frame* pFrame) {
    Frame* pFrameCopy = (Frame*)MEMALLOC(SIZEOF(Frame));
    MEMCPY(pFrameCopy, pFrame, SIZEOF(Frame));
    
    pFrameCopy->frameData = (PBYTE)MEMALLOC(pFrame->size);
    MEMCPY(pFrameCopy->frameData, pFrame->frameData, pFrame->size);
    
    MUTEX_LOCK(pPacker->queueLock);
    queueEnqueue(pPacker->pendingFrames, pFrameCopy);
    MUTEX_UNLOCK(pPacker->queueLock);
    
    return STATUS_SUCCESS;
}

4. 自适应MTU优化

// 自适应MTU检测
typedef struct {
    UINT32 currentMtu;
    UINT32 minMtu;
    UINT32 maxMtu;
    UINT32 probeInterval;
    UINT32 consecutiveFailures;
    UINT32 consecutiveSuccesses;
    MUTEX mtuLock;
} AdaptiveMtuDetector;

STATUS updateAdaptiveMtu(AdaptiveMtuDetector* pDetector, BOOL sendSuccess, UINT32 packetSize) {
    MUTEX_LOCK(pDetector->mtuLock);
    
    if (sendSuccess) {
        pDetector->consecutiveSuccesses++;
        pDetector->consecutiveFailures = 0;
        
        // 成功时尝试增加MTU
        if (pDetector->consecutiveSuccesses >= 10 && 
            pDetector->currentMtu < pDetector->maxMtu) {
            pDetector->currentMtu = MIN(pDetector->currentMtu + 100, pDetector->maxMtu);
            pDetector->consecutiveSuccesses = 0;
            DLOGI("Increased MTU to %u bytes", pDetector->currentMtu);
        }
    } else {
        pDetector->consecutiveFailures++;
        pDetector->consecutiveSuccesses = 0;
        
        // 失败时减少MTU
        if (pDetector->consecutiveFailures >= 3 && 
            pDetector->currentMtu > pDetector->minMtu) {
            pDetector->currentMtu = MAX(pDetector->currentMtu - 50, pDetector->minMtu);
            pDetector->consecutiveFailures = 0;
            DLOGW("Decreased MTU to %u bytes due to send failures", pDetector->currentMtu);
        }
    }
    
    MUTEX_UNLOCK(pDetector->mtuLock);
    return STATUS_SUCCESS;
}

错误处理与恢复

1. 打包错误处理

// RTP打包错误处理
typedef enum {
    RTP_ERROR_NONE = 0,
    RTP_ERROR_INVALID_INPUT,
    RTP_ERROR_BUFFER_TOO_SMALL,
    RTP_ERROR_NALU_BOUNDARY_NOT_FOUND,
    RTP_ERROR_UNSUPPORTED_CODEC,
    RTP_ERROR_FRAGMENTATION_FAILED,
    RTP_ERROR_MEMORY_ALLOCATION_FAILED
} RtpPackError;

STATUS handleRtpPackError(RtpPackError error, PVOID context) {
    switch (error) {
        case RTP_ERROR_INVALID_INPUT:
            DLOGE("Invalid input parameters for RTP packing");
            return STATUS_RTP_INVALID_INPUT;
            
        case RTP_ERROR_BUFFER_TOO_SMALL:
            DLOGE("RTP buffer too small for payload");
            return handleBufferTooSmallError(context);
            
        case RTP_ERROR_NALU_BOUNDARY_NOT_FOUND:
            DLOGE("NAL unit boundary not found in H.264/H.265 data");
            return handleNaluBoundaryError(context);
            
        case RTP_ERROR_UNSUPPORTED_CODEC:
            DLOGE("Unsupported video codec for RTP packing");
            return STATUS_RTP_UNSUPPORTED_CODEC;
            
        case RTP_ERROR_FRAGMENTATION_FAILED:
            DLOGE("RTP fragmentation failed");
            return handleFragmentationError(context);
            
        case RTP_ERROR_MEMORY_ALLOCATION_FAILED:
            DLOGE("Memory allocation failed during RTP packing");
            return STATUS_NOT_ENOUGH_MEMORY;
            
        default:
            DLOGE("Unknown RTP packing error: %d", error);
            return STATUS_RTP_UNKNOWN_ERROR;
    }
}

// 缓冲区太小错误处理
STATUS handleBufferTooSmallError(PVOID context) {
    PRtpPackContext pCtx = (PRtpPackContext)context;
    
    // 重新分配更大的缓冲区
    UINT32 newBufferSize = pCtx->bufferSize * 2;
    PBYTE newBuffer = (PBYTE)MEMALLOC(newBufferSize);
    
    if (newBuffer != NULL) {
        // 复制现有数据
        if (pCtx->buffer != NULL) {
            MEMCPY(newBuffer, pCtx->buffer, pCtx->bufferSize);
            MEMFREE(pCtx->buffer);
        }
        
        pCtx->buffer = newBuffer;
        pCtx->bufferSize = newBufferSize;
        
        DLOGW("RTP buffer resized to %u bytes", newBufferSize);
        return STATUS_SUCCESS;
    }
    
    return STATUS_NOT_ENOUGH_MEMORY;
}

2. 网络错误恢复

// 网络错误恢复策略
typedef struct {
    UINT32 retryCount;
    UINT32 maxRetries;
    UINT32 backoffTimeMs;
    UINT32 maxBackoffTimeMs;
    DOUBLE backoffMultiplier;
    MUTEX recoveryLock;
} NetworkRecoveryManager;

STATUS handleNetworkError(NetworkRecoveryManager* pManager, STATUS errorStatus) {
    MUTEX_LOCK(pManager->recoveryLock);
    
    if (pManager->retryCount >= pManager->maxRetries) {
        DLOGE("Maximum retry count (%u) exceeded, giving up", pManager->maxRetries);
        MUTEX_UNLOCK(pManager->recoveryLock);
        return STATUS_NETWORK_RECOVERY_FAILED;
    }
    
    pManager->retryCount++;
    
    // 计算退避时间
    UINT32 backoffTime = pManager->backoffTimeMs;
    for (UINT32 i = 1; i < pManager->retryCount; i++) {
        backoffTime = (UINT32)(backoffTime * pManager->backoffMultiplier);
    }
    
    backoffTime = MIN(backoffTime, pManager->maxBackoffTimeMs);
    
    DLOGW("Network error recovery attempt %u/%u, backing off for %u ms", 
          pManager->retryCount, pManager->maxRetries, backoffTime);
    
    // 执行退避
    THREAD_SLEEP(backoffTime);
    
    MUTEX_UNLOCK(pManager->recoveryLock);
    return STATUS_SUCCESS;
}

// 重置恢复状态
VOID resetNetworkRecovery(NetworkRecoveryManager* pManager) {
    MUTEX_LOCK(pManager->recoveryLock);
    pManager->retryCount = 0;
    MUTEX_UNLOCK(pManager->recoveryLock);
}

性能监控与统计

1. 打包性能指标

// RTP打包性能统计
typedef struct {
    // 基础统计
    UINT64 totalFramesProcessed;      // 处理的总帧数
    UINT64 totalPacketsGenerated;       // 生成的总包数
    UINT64 totalBytesProcessed;       // 处理的总字节数
    
    // 分片统计
    UINT64 singleNaluPackets;         // 单NAL单元包数
    UINT64 fragmentedPackets;         // 分片包数
    UINT64 aggregatedPackets;           // 聚合包数
    
    // 性能指标
    DOUBLE averagePacketsPerFrame;    // 平均每帧包数
    DOUBLE averageBytesPerPacket;     // 平均每包字节数
    DOUBLE fragmentationRatio;        // 分片比例
    
    // 错误统计
    UINT64 packErrors;                // 打包错误数
    UINT64 bufferOverflows;           // 缓冲区溢出数
    UINT64 codecErrors;               // 编解码错误数
    
    // 时间统计
    UINT64 totalPackTimeUs;           // 总打包时间（微秒）
    DOUBLE averagePackTimeUs;         // 平均打包时间
    UINT64 maxPackTimeUs;             // 最大打包时间
    UINT64 minPackTimeUs;             // 最小打包时间
} RtpPackStatistics;

// 更新打包统计
VOID updateRtpPackStats(RtpPackStatistics* pStats, const Frame* pFrame, 
                       UINT32 packetCount, UINT64 packTimeUs) {
    pStats->totalFramesProcessed++;
    pStats->totalPacketsGenerated += packetCount;
    pStats->totalBytesProcessed += pFrame->size;
    pStats->totalPackTimeUs += packTimeUs;
    
    // 更新分片统计
    if (packetCount == 1) {
        pStats->singleNaluPackets++;
    } else if (packetCount > 1) {
        pStats->fragmentedPackets += packetCount;
    }
    
    // 更新性能指标
    pStats->averagePacketsPerFrame = (DOUBLE)pStats->totalPacketsGenerated / pStats->totalFramesProcessed;
    pStats->averageBytesPerPacket = (DOUBLE)pStats->totalBytesProcessed / pStats->totalPacketsGenerated;
    pStats->fragmentationRatio = (DOUBLE)pStats->fragmentedPackets / pStats->totalPacketsGenerated;
    pStats->averagePackTimeUs = (DOUBLE)pStats->totalPackTimeUs / pStats->totalFramesProcessed;
    
    // 更新时间统计
    if (packTimeUs > pStats->maxPackTimeUs) {
        pStats->maxPackTimeUs = packTimeUs;
    }
    if (pStats->minPackTimeUs == 0 || packTimeUs < pStats->minPackTimeUs) {
        pStats->minPackTimeUs = packTimeUs;
    }
}

2. 实时监控

// 实时性能监控
typedef struct {
    RtpPackStatistics* pStats;
    UINT32 reportIntervalSeconds;
    UINT64 lastReportTime;
    MUTEX statsLock;
} RtpPackMonitor;

VOID rtpPackMonitorReport(RtpPackMonitor* pMonitor) {
    UINT64 currentTime = GETTIME() / HUNDREDS_OF_NANOS_IN_A_SECOND;
    
    if (currentTime - pMonitor->lastReportTime >= pMonitor->reportIntervalSeconds) {
        MUTEX_LOCK(pMonitor->statsLock);
        
        RtpPackStatistics* pStats = pMonitor->pStats;
        
        DLOGI("=== RTP Pack Performance Report ===");
        DLOGI("Frames Processed: %llu", pStats->totalFramesProcessed);
        DLOGI("Packets Generated: %llu", pStats->totalPacketsGenerated);
        DLOGI("Bytes Processed: %llu", pStats->totalBytesProcessed);
        DLOGI("Average Packets/Frame: %.2f", pStats->averagePacketsPerFrame);
        DLOGI("Average Bytes/Packet: %.2f", pStats->averageBytesPerPacket);
        DLOGI("Fragmentation Ratio: %.2f%%", pStats->fragmentationRatio * 100);
        DLOGI("Average Pack Time: %.2f us", pStats->averagePackTimeUs);
        DLOGI("Pack Time Range: [%llu, %llu] us", pStats->minPackTimeUs, pStats->maxPackTimeUs);
        
        if (pStats->packErrors > 0) {
            DLOGW("Pack Errors: %llu", pStats->packErrors);
        }
        
        pMonitor->lastReportTime = currentTime;
        MUTEX_UNLOCK(pMonitor->statsLock);
    }
}

总结

视频文件RTP打包是WebRTC媒体传输的核心技术，涉及多个复杂的处理环节：

1. 核心技术特点

• 编解码器适配：支持H.264、H.265、VP8等多种编码格式
• 智能分片策略：基于MTU大小的自适应分片
• 关键帧优化：优先处理关键帧，确保快速错误恢复
• 零拷贝技术：减少内存拷贝，提高性能

2. 性能优化策略

• 内存池管理：预分配缓冲区，减少动态分配
• 并行处理：多线程并行打包，提高吞吐量
• 自适应MTU：根据网络状况动态调整包大小
• 硬件加速：利用SIMD指令加速数据处理

3. 错误处理机制

• 多层次错误检测：从编解码到网络传输的全链路错误处理
• 自动恢复策略：网络错误时的自动重传和退避机制
• 降级处理：在资源受限情况下的优雅降级

4. 监控与调优

• 实时性能监控：关键指标的实时收集和报告
• 自适应调优：基于性能数据的参数自动调整
• A/B测试支持：不同打包策略的效果对比

通过精密的打包机制和持续的性能优化，WebRTC能够在各种网络环境下提供高质量的视频传输服务，满足不同应用场景的需求。

参考资源

• RFC 6184 - RTP Payload Format for H.264 Video
• RFC 7798 - RTP Payload Format for H.265 Video
• RFC 7741 - RTP Payload Format for VP8 Video
• H.264/AVC Video Coding Standard
• H.265/HEVC Video Coding Standard
• VP8 Video Codec Specification