Temporally Aligned Audio for Video with Autoregression

TL;DR

V-AURA is an autoregressive model that achieves precise temporal alignment and relevance in video-to-audio generation by leveraging high-framerate visual features and a new benchmark dataset, VisualSound.

Contribution

The paper introduces V-AURA, the first autoregressive model for high-quality, temporally aligned video-to-audio synthesis, and presents VisualSound, a curated dataset for evaluating audio-visual relevance.

Findings

V-AURA outperforms existing models in temporal alignment.

V-AURA maintains high audio quality comparable to state-of-the-art.

VisualSound dataset improves evaluation of audio-visual relevance.

Abstract

We introduce V-AURA, the first autoregressive model to achieve high temporal alignment and relevance in video-to-audio generation. V-AURA uses a high-framerate visual feature extractor and a cross-modal audio-visual feature fusion strategy to capture fine-grained visual motion events and ensure precise temporal alignment. Additionally, we propose VisualSound, a benchmark dataset with high audio-visual relevance. VisualSound is based on VGGSound, a video dataset consisting of in-the-wild samples extracted from YouTube. During the curation, we remove samples where auditory events are not aligned with the visual ones. V-AURA outperforms current state-of-the-art models in temporal alignment and semantic relevance while maintaining comparable audio quality. Code, samples, VisualSound and models are available at https://v-aura.notion.site