AC/DC: LLM-based Audio Comprehension via Dialogue Continuation

TL;DR

This paper introduces AC/DC, an LLM-based audio comprehension model that uses dialogue continuation training to improve zero-shot instruction-following in audio understanding tasks, without requiring multitask tuning.

Contribution

It presents a novel dialogue continuation training approach for audio comprehension models, enabling zero-shot instruction-following from audio captioning data alone.

Findings

Effective zero-shot instruction-following demonstrated on multiple datasets.

Outperforms baseline models in audio comprehension tasks.

Mitigates caption variation issues through dialogue-based training.

Abstract

We propose an instruction-following audio comprehension model that leverages the dialogue continuation ability of large language models (LLMs). Instead of directly generating target captions in training data, the proposed method trains a model to produce responses as if the input caption triggered a dialogue. This dialogue continuation training mitigates the caption variation problem. Learning to continue a dialogue effectively captures the caption's meaning beyond its surface-level words. As a result, our model enables zero-shot instruction-following capability without multitask instruction tuning, even trained solely on audio captioning datasets. Experiments on AudioCaps, WavCaps, and Clotho datasets with AudioBench audio-scene question-answering tests demonstrate our model's ability to follow various unseen instructions.