Rhythmic Foley: A Framework For Seamless Audio-Visual Alignment In Video-to-Audio Synthesis

TL;DR

This paper presents Rhythmic Foley, a novel framework for video-to-audio synthesis that enhances synchronization and semantic accuracy using dual adapters and contrastive pre-training.

Contribution

It introduces a dual-adapter framework with semantic and temporal synchronization, improving audio-visual alignment and control in video-to-audio synthesis.

Findings

Improved semantic integrity in generated audio.

Enhanced beat point synchronization accuracy.

Effective control over audio semantics and beat effects.

Abstract

Our research introduces an innovative framework for video-to-audio synthesis, which solves the problems of audio-video desynchronization and semantic loss in the audio. By incorporating a semantic alignment adapter and a temporal synchronization adapter, our method significantly improves semantic integrity and the precision of beat point synchronization, particularly in fast-paced action sequences. Utilizing a contrastive audio-visual pre-trained encoder, our model is trained with video and high-quality audio data, improving the quality of the generated audio. This dual-adapter approach empowers users with enhanced control over audio semantics and beat effects, allowing the adjustment of the controller to achieve better results. Extensive experiments substantiate the effectiveness of our framework in achieving seamless audio-visual alignment.