Coherent Audio-Visual Editing via Conditional Audio Generation Following Video Edits

TL;DR

This paper presents a new pipeline for joint audio-visual editing that ensures coherence between edited videos and their audio by leveraging a conditional audio generation model and innovative training strategies.

Contribution

It introduces a novel video-to-audio generation model conditioned on visual edits, along with a data augmentation strategy to improve training efficiency and audio-visual coherence.

Findings

Outperforms existing methods in audio-visual alignment

Maintains content integrity after edits

Effectively adapts to complex visual changes

Abstract

We introduce a novel pipeline for joint audio-visual editing that enhances the coherence between edited video and its accompanying audio. Our approach first applies state-of-the-art video editing techniques to produce the target video, then performs audio editing to align with the visual changes. To achieve this, we present a new video-to-audio generation model that conditions on the source audio, target video, and a text prompt. We extend the model architecture to incorporate conditional audio input and propose a data augmentation strategy that improves training efficiency. Furthermore, our model dynamically adjusts the influence of the source audio based on the complexity of the edits, preserving the original audio structure where possible. Experimental results demonstrate that our method outperforms existing approaches in maintaining audio-visual alignment and content integrity.