TAC: Timestamped Audio Captioning
Sonal Kumar, Prem Seetharaman, Ke Chen, Oriol Nieto, Jiaqi Su, Zhepei Wang, Rithesh Kumar, Dinesh Manocha, Nicholas J. Bryan, Zeyu Jin, Justin Salamon

TL;DR
This paper introduces TAC, a model for generating temporally grounded audio descriptions in complex scenes, improving accuracy and reducing hallucinations, and extends it with TAC-V for audio-visual descriptions, enhancing multimodal understanding.
Contribution
The paper presents TAC, a novel timestamped audio captioning model trained on synthetic data, and TAC-V, an audio-visual extension, serving as semantic bridges for improved reasoning benchmarks.
Findings
TAC outperforms existing methods in event detection and dense captioning.
TAC and TAC-V achieve state-of-the-art scores on multiple audio and audio-visual benchmarks.
The models exhibit low hallucination rates and accurate temporal grounding.
Abstract
Large Audio Language Models struggle to disentangle overlapping events in complex acoustic scenes, yielding temporally inconsistent captions and frequent hallucinations. We introduce Timestamped Audio Captioner (TAC), a model that produces temporally grounded audio descriptions at varying degrees of detail and resolution. TAC is trained with a synthetic data pipeline that constructs challenging and dynamic mixtures from real-world audio sources, enabling robust learning under realistic polyphonic conditions. Across event detection and dense captioning, TAC outperforms all competing methods, with a low hallucination rate and accurate temporal grounding. We also introduce TAC-V, an audio-visual pipeline to generate semantically rich audio-visual descriptions. We then show that TAC and TAC-V serves as a "semantic bridge" for a text-only reasoner: a simple TACLLM and…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsMusic and Audio Processing · Multimodal Machine Learning Applications · Speech and Audio Processing
