Smooth-Foley: Creating Continuous Sound for Video-to-Audio Generation Under Semantic Guidance

TL;DR

Smooth-Foley is a novel video-to-audio generation model that uses semantic guidance from text to improve audio-video alignment and produce continuous, high-quality Foley sounds, especially in challenging scenarios with moving visuals.

Contribution

It introduces a semantic-guided generative model with specialized adapters to enhance temporal and semantic alignment in video-to-audio synthesis.

Findings

Outperforms existing models in continuous sound scenarios

Achieves higher quality and better physical law adherence

Improves semantic and temporal alignment in audio generation

Abstract

The video-to-audio (V2A) generation task has drawn attention in the field of multimedia due to the practicality in producing Foley sound. Semantic and temporal conditions are fed to the generation model to indicate sound events and temporal occurrence. Recent studies on synthesizing immersive and synchronized audio are faced with challenges on videos with moving visual presence. The temporal condition is not accurate enough while low-resolution semantic condition exacerbates the problem. To tackle these challenges, we propose Smooth-Foley, a V2A generative model taking semantic guidance from the textual label across the generation to enhance both semantic and temporal alignment in audio. Two adapters are trained to leverage pre-trained text-to-audio generation models. A frame adapter integrates high-resolution frame-wise video features while a temporal adapter integrates temporal conditions obtained from similarities of visual frames and textual labels. The incorporation of semantic guidance from textual labels achieves precise audio-video alignment. We conduct extensive quantitative and qualitative experiments. Results show that Smooth-Foley performs better than existing models on both continuous sound scenarios and general scenarios. With semantic guidance, the audio generated by Smooth-Foley exhibits higher quality and better adherence to physical laws.