AVControl: Efficient Framework for Training Audio-Visual Controls

TL;DR

AVControl is a modular, efficient framework for training diverse audio-visual controls on joint models, outperforming existing methods on multiple benchmarks with minimal data and computational resources.

Contribution

Introduces AVControl, a lightweight, extendable framework that trains separate control modalities as LoRA adapters on a joint audio-visual foundation model, enabling diverse controls without architectural changes.

Findings

Outperforms baselines on depth- and pose-guided generation

Achieves competitive results on camera control and audio-visual benchmarks

Supports a wide range of independently trained modalities

Abstract

Controlling video and audio generation requires diverse modalities, from depth and pose to camera trajectories and audio transformations, yet existing approaches either train a single monolithic model for a fixed set of controls or introduce costly architectural changes for each new modality. We introduce AVControl, a lightweight, extendable framework built on LTX-2, a joint audio-visual foundation model, where each control modality is trained as a separate LoRA on a parallel canvas that provides the reference signal as additional tokens in the attention layers, requiring no architectural changes beyond the LoRA adapters themselves. We show that simply extending image-based in-context methods to video fails for structural control, and that our parallel canvas approach resolves this. On the VACE Benchmark, we outperform all evaluated baselines on depth- and pose-guided generation, inpainting, and outpainting, and show competitive results on camera control and audio-visual benchmarks. Our framework supports a diverse set of independently trained modalities: spatially-aligned controls such as depth, pose, and edges, camera trajectory with intrinsics, sparse motion control, video editing, and, to our knowledge, the first modular audio-visual controls for a joint generation model. Our method is both compute- and data-efficient: each modality requires only a small dataset and converges within a few hundred to a few thousand training steps, a fraction of the budget of monolithic alternatives. We publicly release our code and trained LoRA checkpoints.