Low-Resource Guidance for Controllable Latent Audio Diffusion

TL;DR

This paper introduces a low-resource, guidance-based method for controllable latent audio diffusion that reduces computational costs by operating directly in latent space, enabling effective control over audio attributes with minimal training.

Contribution

The authors propose LatCHs, a novel approach that controls latent audio diffusion models efficiently without decoder backpropagation, requiring minimal training resources.

Findings

Effective control over audio attributes like pitch and beats.

Maintains high audio quality with lower computational costs.

Operates with minimal training parameters and time.

Abstract

Generative audio requires fine-grained controllable outputs, yet most existing methods require model retraining on specific controls or inference-time controls (\textit{e.g.}, guidance) that can also be computationally demanding. By examining the bottlenecks of existing guidance-based controls, in particular their high cost-per-step due to decoder backpropagation, we introduce a guidance-based approach through selective TFG and Latent-Control Heads (LatCHs), which enables controlling latent audio diffusion models with low computational overhead. LatCHs operate directly in latent space, avoiding the expensive decoder step, and requiring minimal training resources (7M parameters and $\approx$ 4 hours of training). Experiments with Stable Audio Open demonstrate effective control over intensity, pitch, and beats (and a combination of those) while maintaining generation quality. Our method balances precision and audio fidelity with far lower computational costs than standard end-to-end guidance. Demo examples can be found at https://zacharynovack.github.io/latch/latch.html.