TimeChat-Captioner: Scripting Multi-Scene Videos with Time-Aware and Structural Audio-Visual Captions

TL;DR

This paper introduces Omni Dense Captioning, a new task for generating detailed, structured, and time-aware audio-visual narratives with a comprehensive benchmark and a strong baseline model, advancing video captioning and reasoning capabilities.

Contribution

It proposes a novel task, Omni Dense Captioning, along with a structured schema, benchmark dataset, evaluation metric, and a powerful baseline model trained for dense, time-aware video captioning.

Findings

The baseline model achieves state-of-the-art performance.

Dense descriptions improve audio-visual reasoning.

Structured captions enhance temporal grounding.

Abstract

This paper proposes Omni Dense Captioning, a novel task designed to generate continuous, fine-grained, and structured audio-visual narratives with explicit timestamps. To ensure dense semantic coverage, we introduce a six-dimensional structural schema to create "script-like" captions, enabling readers to vividly imagine the video content scene by scene, akin to a cinematographic screenplay. To facilitate research, we construct OmniDCBench, a high-quality, human-annotated benchmark, and propose SodaM, a unified metric that evaluates time-aware detailed descriptions while mitigating scene boundary ambiguity. Furthermore, we construct a training dataset, TimeChatCap-42K, and present TimeChat-Captioner-7B, a strong baseline trained via SFT and GRPO with task-specific rewards. Extensive experiments demonstrate that TimeChat-Captioner-7B achieves state-of-the-art performance, surpassing Gemini-2.5-Pro, while its generated dense descriptions significantly boost downstream capabilities in audio-visual reasoning (DailyOmni and WorldSense) and temporal grounding (Charades-STA). All datasets, models, and code will be made publicly available at https://github.com/yaolinli/TimeChat-Captioner.