Scaling Up Audio-Synchronized Visual Animation: An Efficient Training Paradigm

TL;DR

This paper introduces an efficient two-stage training method for audio-synchronized visual animation that leverages noisy videos for pretraining and minimal high-quality data for fine-tuning, enabling scalable and diverse animation generation.

Contribution

The authors propose a novel two-stage training paradigm that reduces manual curation by over 10 times and enhances scalability to diverse classes using noisy videos and minimal high-quality data.

Findings

Significantly reduces manual curation effort.

Generalizes well to diverse and open-world classes.

Achieves improved synchronization with multi-feature conditioning.

Abstract

Recent advances in audio-synchronized visual animation enable control of video content using audios from specific classes. However, existing methods rely heavily on expensive manual curation of high-quality, class-specific training videos, posing challenges to scaling up to diverse audio-video classes in the open world. In this work, we propose an efficient two-stage training paradigm to scale up audio-synchronized visual animation using abundant but noisy videos. In stage one, we automatically curate large-scale videos for pretraining, allowing the model to learn diverse but imperfect audio-video alignments. In stage two, we finetune the model on manually curated high-quality examples, but only at a small scale, significantly reducing the required human effort. We further enhance synchronization by allowing each frame to access rich audio context via multi-feature conditioning and window attention. To efficiently train the model, we leverage pretrained text-to-video generator and audio encoders, introducing only 1.9\% additional trainable parameters to learn audio-conditioning capability without compromising the generator's prior knowledge. For evaluation, we introduce AVSync48, a benchmark with videos from 48 classes, which is 3$\times$ more diverse than previous benchmarks. Extensive experiments show that our method significantly reduces reliance on manual curation by over 10$\times$, while generalizing to many open classes.