TAC: Timestamped Audio Captioning

TL;DR

This paper introduces TAC, a model for generating temporally grounded audio descriptions in complex scenes, improving accuracy and reducing hallucinations, and extends it with TAC-V for audio-visual descriptions, enhancing multimodal understanding.

Contribution

The paper presents TAC, a novel timestamped audio captioning model trained on synthetic data, and TAC-V, an audio-visual extension, serving as semantic bridges for improved reasoning benchmarks.

Findings

TAC outperforms existing methods in event detection and dense captioning.

TAC and TAC-V achieve state-of-the-art scores on multiple audio and audio-visual benchmarks.

The models exhibit low hallucination rates and accurate temporal grounding.

Abstract

Large Audio Language Models struggle to disentangle overlapping events in complex acoustic scenes, yielding temporally inconsistent captions and frequent hallucinations. We introduce Timestamped Audio Captioner (TAC), a model that produces temporally grounded audio descriptions at varying degrees of detail and resolution. TAC is trained with a synthetic data pipeline that constructs challenging and dynamic mixtures from real-world audio sources, enabling robust learning under realistic polyphonic conditions. Across event detection and dense captioning, TAC outperforms all competing methods, with a low hallucination rate and accurate temporal grounding. We also introduce TAC-V, an audio-visual pipeline to generate semantically rich audio-visual descriptions. We then show that TAC and TAC-V serves as a "semantic bridge" for a text-only reasoner: a simple TAC$\rightarrow$LLM and TAC-V$\rightarrow$LLM cascade achieves state-of-the-art scores on benchmarks for both audio (MMAU-Pro, MMSU, MMAR) and audio-visual (DailyOmni, VideoHolmes) understanding and reasoning respectively.