AudioComposer: Towards Fine-grained Audio Generation with Natural Language Descriptions

TL;DR

AudioComposer introduces a natural language-driven framework for fine-grained audio generation, effectively controlling content and style without complex reference conditions, and surpassing existing models in quality and controllability.

Contribution

The paper presents a novel TTA framework using only natural language descriptions and flow-based diffusion transformers, along with a data simulation pipeline to improve fine-grained control and data availability.

Findings

Outperforms state-of-the-art TTA models in quality and controllability.

Uses a flow-based diffusion transformer with cross-attention for effective text integration.

Reduces model size while maintaining superior performance.

Abstract

Current Text-to-audio (TTA) models mainly use coarse text descriptions as inputs to generate audio, which hinders models from generating audio with fine-grained control of content and style. Some studies try to improve the granularity by incorporating additional frame-level conditions or control networks. However, this usually leads to complex system design and difficulties due to the requirement for reference frame-level conditions. To address these challenges, we propose AudioComposer, a novel TTA generation framework that relies solely on natural language descriptions (NLDs) to provide both content specification and style control information. To further enhance audio generative modeling, we employ flow-based diffusion transformers with the cross-attention mechanism to incorporate text descriptions effectively into audio generation processes, which can not only simultaneously consider the content and style information in the text inputs, but also accelerate generation compared to other architectures. Furthermore, we propose a novel and comprehensive automatic data simulation pipeline to construct data with fine-grained text descriptions, which significantly alleviates the problem of data scarcity in the area. Experiments demonstrate the effectiveness of our framework using solely NLDs as inputs for content specification and style control. The generation quality and controllability surpass state-of-the-art TTA models, even with a smaller model size.