D-ORCA: Dialogue-Centric Optimization for Robust Audio-Visual Captioning

TL;DR

D-ORCA is a dialogue-centric, multimodal large language model designed for robust audio-visual captioning, utilizing a new bilingual dataset and novel reinforcement learning rewards to improve speaker attribution, speech recognition, and temporal grounding.

Contribution

Introduces D-ORCA, a novel dialogue-centric multimodal LLM with a large bilingual dataset and reinforcement learning rewards for improved audio-visual captioning.

Findings

D-ORCA outperforms existing open-source models in key tasks.

Achieves performance comparable to larger models like Qwen3-Omni.

Effectively utilizes reinforcement learning rewards for captioning accuracy.

Abstract

Spoken dialogue is a primary source of information in videos; therefore, accurately identifying who spoke what and when is essential for deep video understanding. We introduce D-ORCA, a \textbf{d}ialogue-centric \textbf{o}mni-modal large language model optimized for \textbf{r}obust audio-visual \textbf{ca}ptioning. We further curate DVD, a large-scale, high-quality bilingual dataset comprising nearly 40,000 multi-party dialogue videos for training and 2000 videos for evaluation in English and Mandarin, addressing a critical gap in the open-source ecosystem. To ensure fine-grained captioning accuracy, we adopt group relative policy optimization with three novel reward functions that assess speaker attribution accuracy, global speech content accuracy, and sentence-level temporal boundary alignment. These rewards are derived from evaluation metrics widely used in speech processing and, to our knowledge, are applied for the first time as reinforcement learning objectives for audio-visual captioning. Extensive experiments demonstrate that D-ORCA substantially outperforms existing open-source models in speaker identification, speech recognition, and temporal grounding. Notably, despite having only 8 billion parameters, D-ORCA achieves performance competitive with Qwen3-Omni across several general-purpose audio-visual understanding benchmarks. Demos are available at \href{https://d-orca-llm.github.io/}{https://d-orca-llm.github.io/}. Our code, data, and checkpoints will be available at \href{https://github.com/WeChatCV/D-ORCA/}{https://github.com/WeChatCV/D-ORCA/}.