Qt 中的 WebRTC 视频：抖动缓冲区 + NACK

在Qt客户端中实现WebRTC视频的抖动缓冲与NACK机制

WebRTC采用UDP协议以最小化传输开销。客户端交换SDP描述符，使用STUN/TURN进行NAT穿透，通过ICE保持连接，并借助信令服务器完成协调。SFU/MCU服务器作为中间盒负责流转发。一个1080p@30fps的VP8码流在无关键帧情况下约需600 Kbps。丢包会因PLI请求和NACK机制显著增加负载，而抖动缓冲则有效解决数据包乱序问题。

核心任务包括：支持NACK的信令/MCU服务器、重传机制以及客户端侧的抖动缓冲。

使用NACK支持的Pion服务器

服务器基于Go语言，使用Pion库实现。我们注册了默认的拦截器以支持NACK：

type Controller interface {
	HandleConnection(c *common.SafeWebSocket)
	JoinRoom(peer *common.Peer, msg Msg) error
	LeaveRoom(peer *common.Peer, msg Msg) error
}

type controller struct {
	logger *zap.Logger

	roomRepo repository.RoomRepo
	api      *webrtc.API
}

func NewController(logger *zap.Logger, roomRepo repository.RoomRepo) Controller {
	settingEngine := webrtc.SettingEngine{}
	settingEngine.SetAnsweringDTLSRole(webrtc.DTLSRoleServer)
	mediaEngine := &webrtc.MediaEngine{}
	mediaEngine.RegisterDefaultCodecs()
	interseporRegistry := interceptor.Registry{}

	if err := webrtc.RegisterDefaultInterceptorsWithOptions(mediaEngine, &interseporRegistry,
		webrtc.WithNackGeneratorOptions(nack.GeneratorSize(8192)),
		webrtc.WithNackResponderOptions(nack.ResponderSize(8192)),
	); err != nil {
		logger.Error("failed to register interceptor", zap.Error(err))

		panic(err)
	}

	api := webrtc.NewAPI(
		webrtc.WithMediaEngine(mediaEngine),
		webrtc.WithSettingEngine(settingEngine),
		webrtc.WithInterceptorRegistry(&interseporRegistry),
	)

	ctrl := &controller{
		api:      api,
		logger:   logger,
		roomRepo: roomRepo,
	}

	go func() { // 每两秒发送一次RTCP I帧请求
		ticker := time.NewTicker(2 * time.Second)
		for _ = range ticker.C {
			roomIds := ctrl.roomRepo.GetRooms()
			for _, roomId := range roomIds {
				go ctrl.dispatch(roomId)
			}
		}
	}()

	return ctrl
}

网络模拟丢包：

sudo tc qdisc add dev lo root netem delay 50ms 20ms loss 1%

使用libdatachannel的Qt客户端

浏览器通过轻量级HTTP服务器充当视频源。C++客户端使用libdatachannel + Qt，无需完整Chromium环境。

建立对等连接：

void ConferenceClient::connectClient(QString url, QString roomId)
{
    rtc::InitLogger(rtc::LogLevel::Debug);

    this->pc.onLocalDescription(this->pcOnLocalDescription(roomId));
    this->pc.onLocalCandidate(this->pcOnLocalCandidate());
    this->pc.onGatheringStateChange(this->pcOnGatheringStateChange());

    this->pc.onIceStateChange( {
        std::cout << "Ice state changed: " << state << std::endl;
    });
    this->pc.onStateChange( {
        std::cout << "state changed: " << state << std::endl;
    });

    this->ws.onOpen(this->wsOnOpen(roomId));
    this->ws.onMessage(this->wsOnMessage());

    this->pc.onTrack(this->pcOnTrack());
    this->ws.open(url.toStdString());
}

处理音视频轨道与数据包

为每个轨道创建结构体，包含抖动缓冲（LRUCache）和帧队列：

std::function<void(std::shared_ptr<rtc::Track>)> ConferenceClient::pcOnTrack() {
    return this {
        auto mid = track->description().mid();

        this->track_index[mid]
            = {track, 0, "NO_VALUE", 0, 0, LRUCache<std::uint32_t, jitterbuffer>(256)};

        this->player->initMid(mid);
        bool isVideo = true;

        if (track->description().type() == "audio") {
            isVideo = false;

            track->setMediaHandler(std::make_shared<rtc::OpusRtpDepacketizer>());
            track->chainMediaHandler(std::make_shared<rtc::RtcpReceivingSession>());
            track->onFrame(this->trackOnFrame(mid, isVideo));
        } else {
            track->onMessage(this->pcOnMessage(mid));
        }

        track->onOpen([track]() { track->requestKeyframe(); });
        track->onClosed([this, mid]() { this->player->destroy(mid); });
    };
}

RTP数据包处理的关键点：

• 丢弃延迟到达的数据包（rtpHeader->timestamp() < lastCompletedTs）
• RTX重传——恢复原始seqNumber和payloadType
• 字节序问题——RTP数据为大端格式，平台为小端
• 帧队列使用std::map<uint32_t, pair<long, vector<byte>>>，以RTP时间戳为索引
• 播放时机：在首个帧数据包到达后等待PLAYER_DELAY再开始播放

重组VP8/VP9帧

std::function<void(rtc::message_variant)> ConferenceClient::pcOnMessage(std::string mid)
{
    return this, mid {
        // ... 时间戳检查、RTX处理 ...
        
        std::vector<std::byte> frame;

        if (!frame_cache.exist(pkgTs)
            && (track->description().rtpMap(PT)->format == MyApp::VP8CODEC
                || track->description().rtpMap(PT)->format == MyApp::VP9CODEC)) {
            frame_cache.put(pkgTs, jitterbuffer());

            track_info.ssrc = rtpHeader->ssrc();

            track_info.frame_queue[pkgTs] = std::make_pair(nowTs, std::vector<std::byte>());

            codec = track->description().rtpMap(PT)->format;
            codecPT = PT;
        }

        jitterbuffer &buff = frame_cache.get(pkgTs);

        if (codec == MyApp::VP9CODEC) {
            frame = buff.addVp9Packet(std::move(msg), track_info.lastCompletedTs);
        } else if (codec == MyApp::VP8CODEC) {
            frame = buff.addVp8Packet(std::move(msg), track_info.lastCompletedTs);
        }

        if (frame.size() > 0) {
            track_info.frame_queue[pkgTs].second = std::move(frame);
        }
        // ... 播放与NACK处理 ...
    };
}

核心要点总结

• libdatachannel需手动实现抖动缓冲与NACK机制
• Pion拦截器可自动处理NACK，缓冲区大小设为8192
• 帧队列使用std::map，按RTP时间戳自然排序
• RTX映射在SDP协商阶段确定
• 播放在首帧到达后延迟PLAYER_DELAY才开始
• 每2秒周期性发送一次RTCP PLI请求

— Editorial Team