TADA! Tuning Audio Diffusion Models through Activation Steering

TL;DR

This paper reveals a semantic bottleneck in audio diffusion models and introduces activation steering as a novel, effective method for fine-grained musical attribute control, outperforming existing techniques.

Contribution

The study uncovers a shared attention layer bottleneck and proposes activation steering, establishing a new state-of-the-art in audio concept modulation.

Findings

Activation patching reveals a semantic bottleneck in attention layers.

Activation steering outperforms prompt, score-space, and weight-space methods.

Extensive user study supports the effectiveness of activation steering.

Abstract

Audio diffusion models can synthesize high-fidelity music from text, yet achieving fine-grained control over specific musical attributes remains challenging, as their internal mechanisms for representing high-level concepts are poorly understood. In this work, we use activation patching to demonstrate that recent audio diffusion architectures exhibit a semantic bottleneck, where a small, shared subset of consecutive attention layers controls distinct musical concepts, such as the presence of specific instruments, vocals, or genres. Building on this, we systematically evaluate a broad spectrum of steering paradigms, comparing activation steering against prompt-level, score-space, and weight-space interventions, analyzing the interaction between the steering mechanism and the intervention site. Our new benchmark, supported by an extensive user study, demonstrates that localized activation steering establishes a new state-of-the-art in audio concept modulation.