STA-V2A: Video-to-Audio Generation with Semantic and Temporal Alignment

TL;DR

STA-V2A introduces a novel video-to-audio generation method that leverages semantic and temporal alignment to produce more coherent and synchronized audio from videos, surpassing existing models in quality and consistency.

Contribution

The paper presents a new approach combining local temporal and global semantic video features with cross-modal guidance and introduces a novel metric for audio-temporal alignment.

Findings

Outperforms existing Video-to-Audio models in quality and alignment

Effective use of onset prediction and attentive pooling modules

Validated by subjective and objective evaluations

Abstract

Visual and auditory perception are two crucial ways humans experience the world. Text-to-video generation has made remarkable progress over the past year, but the absence of harmonious audio in generated video limits its broader applications. In this paper, we propose Semantic and Temporal Aligned Video-to-Audio (STA-V2A), an approach that enhances audio generation from videos by extracting both local temporal and global semantic video features and combining these refined video features with text as cross-modal guidance. To address the issue of information redundancy in videos, we propose an onset prediction pretext task for local temporal feature extraction and an attentive pooling module for global semantic feature extraction. To supplement the insufficient semantic information in videos, we propose a Latent Diffusion Model with Text-to-Audio priors initialization and cross-modal guidance. We also introduce Audio-Audio Align, a new metric to assess audio-temporal alignment. Subjective and objective metrics demonstrate that our method surpasses existing Video-to-Audio models in generating audio with better quality, semantic consistency, and temporal alignment. The ablation experiment validated the effectiveness of each module. Audio samples are available at https://y-ren16.github.io/STAV2A.