AudioChat: Unified Audio Storytelling, Editing, and Understanding with Transfusion Forcing

TL;DR

AudioChat is a novel framework that enables unified generation, editing, and understanding of complex multi-source audio stories using a new training paradigm and evaluation metrics, advancing audio foundation modeling.

Contribution

The paper introduces AudioChat, a new paradigm with Audio Transfusion Forcing and dialogue-based training for comprehensive audio story processing.

Findings

Effective multi-source audio story generation and editing.

New metrics for task-specific evaluation.

Demonstrated capabilities via a public demo.

Abstract

Despite recent breakthroughs, audio foundation models struggle in processing complex multi-source acoustic scenes. We refer to this challenging domain as audio stories, which can have multiple speakers and background/foreground sound effects. Compared to traditional audio processing tasks, audio stories introduce new layers of semantic, temporal, and physical complexity. To address this challenge, we propose AudioChat, a framework for developing audio foundation models that can generate, edit, and understand audio stories. AudioChat introduces a new paradigm in which LLM-based toolcalling agents simulate interactions between users and the system, and these simulated dialogues are used as training data. We also introduce a novel Audio Transfusion Forcing objective to train the AudioChat model, allowing it to simultaneously decompose high-level instructions via structured chain-of-thought reasoning and perform interactive multi-turn audio understanding/generation. To evaluate generation and editing performance, we develop three new metrics that directly measure task performance instead of relying upon distribution-based scoring. We highly encourage readers to visit our demo to better understand the capabilities of AudioChat: https://wanchichen.github.io/audiochat/.