AV-Link: Temporally-Aligned Diffusion Features for Cross-Modal Audio-Video Generation

TL;DR

AV-Link introduces a unified diffusion-based framework for cross-modal audio-video generation, significantly improving synchronization by leveraging frozen diffusion models and a novel Fusion Block for bidirectional feature exchange.

Contribution

It presents a novel unified framework that performs both video-to-audio and audio-to-video generation using frozen diffusion models and a Fusion Block for cross-modal alignment.

Findings

Outperforms baseline models in audio-video synchronization

Achieves both A2V and V2A tasks within a single framework

Demonstrates superior results in automatic and subjective evaluations

Abstract

We propose AV-Link, a unified framework for Video-to-Audio (A2V) and Audio-to-Video (A2V) generation that leverages the activations of frozen video and audio diffusion models for temporally-aligned cross-modal conditioning. The key to our framework is a Fusion Block that facilitates bidirectional information exchange between video and audio diffusion models through temporally-aligned self attention operations. Unlike prior work that uses dedicated models for A2V and V2A tasks and relies on pretrained feature extractors, AV-Link achieves both tasks in a single self-contained framework, directly leveraging features obtained by the complementary modality (i.e. video features to generate audio, or audio features to generate video). Extensive automatic and subjective evaluations demonstrate that our method achieves a substantial improvement in audio-video synchronization, outperforming more expensive baselines such as the MovieGen video-to-audio model.