DiaDem: Advancing Dialogue Descriptions in Audiovisual Video Captioning for Multimodal Large Language Models

TL;DR

DiaDem is a novel audiovisual captioning model that significantly improves dialogue description accuracy in video captions, addressing a key challenge in multimodal understanding and generation tasks.

Contribution

The paper introduces DiaDem, a new model with a specialized dataset and benchmark for enhanced dialogue description in audiovisual captioning, outperforming existing models.

Findings

DiaDem outperforms Gemini series in dialogue accuracy

Extensive experiments show room for improvement in commercial models

DiaDem achieves competitive results on general captioning benchmarks

Abstract

Accurate dialogue description in audiovisual video captioning is crucial for downstream understanding and generation tasks. However, existing models generally struggle to produce faithful dialogue descriptions within audiovisual captions. To mitigate this limitation, we propose DiaDem, a powerful audiovisual video captioning model capable of generating captions with more precise dialogue descriptions while maintaining strong overall performance. We first synthesize a high-quality dataset for SFT, then employ a difficulty-partitioned two-stage GRPO strategy to further enhance dialogue descriptions. To enable systematic evaluation of dialogue description capabilities, we introduce DiaDemBench, a comprehensive benchmark designed to evaluate models across diverse dialogue scenarios, emphasizing both speaker attribution accuracy and utterance transcription fidelity in audiovisual captions. Extensive experiments on DiaDemBench reveal even commercial models still exhibit substantial room for improvement in dialogue-aware captioning. Notably, DiaDem not only outperforms the Gemini series in dialogue description accuracy but also achieves competitive performance on general audiovisual captioning benchmarks, demonstrating its overall effectiveness.