DegDiT: Controllable Audio Generation with Dynamic Event Graph Guided Diffusion Transformer

TL;DR

DegDiT is a novel framework that uses dynamic event graphs and diffusion transformers to enable precise, controllable, open-vocabulary text-to-audio generation with improved accuracy and diversity.

Contribution

It introduces a dynamic event graph-guided diffusion transformer and a quality-balanced data pipeline for enhanced controllable audio synthesis.

Findings

Achieves state-of-the-art results on multiple datasets.

Demonstrates superior control over temporal and semantic aspects.

Provides diverse and high-quality audio outputs.

Abstract

Controllable text-to-audio generation aims to synthesize audio from textual descriptions while satisfying user-specified constraints, including event types, temporal sequences, and onset and offset timestamps. This enables precise control over both the content and temporal structure of the generated audio. Despite recent progress, existing methods still face inherent trade-offs among accurate temporal localization, open-vocabulary scalability, and practical efficiency. To address these challenges, we propose DegDiT, a novel dynamic event graph-guided diffusion transformer framework for open-vocabulary controllable audio generation. DegDiT encodes the events in the description as structured dynamic graphs. The nodes in each graph are designed to represent three aspects: semantic features, temporal attributes, and inter-event connections. A graph transformer is employed to integrate these nodes and produce contextualized event embeddings that serve as guidance for the diffusion model. To ensure high-quality and diverse training data, we introduce a quality-balanced data selection pipeline that combines hierarchical event annotation with multi-criteria quality scoring, resulting in a curated dataset with semantic diversity. Furthermore, we present consensus preference optimization, facilitating audio generation through consensus among multiple reward signals. Extensive experiments on AudioCondition, DESED, and AudioTime datasets demonstrate that DegDiT achieves state-of-the-art performances across a variety of objective and subjective evaluation metrics.