Schrodinger Audio-Visual Editor: Object-Level Audiovisual Removal

TL;DR

This paper introduces SAVEBench, a new dataset and the Schrodinger Audio-Visual Editor (SAVE), a model that jointly edits audio and visual content at the object level, enabling precise removal while maintaining synchronization and semantic alignment.

Contribution

The paper presents SAVEBench dataset and a novel end-to-end flow-matching model, SAVE, for joint audiovisual editing using a Schrodinger Bridge for direct source-to-target transformation.

Findings

SAVE effectively removes target objects from audio and video.

SAVE achieves stronger temporal synchronization.

SAVE maintains audiovisual semantic correspondence.

Abstract

Joint editing of audio and visual content is crucial for precise and controllable content creation. This new task poses challenges due to the limitations of paired audio-visual data before and after targeted edits, and the heterogeneity across modalities. To address the data and modeling challenges in joint audio-visual editing, we introduce SAVEBench, a paired audiovisual dataset with text and mask conditions to enable object-grounded source-to-target learning. With SAVEBench, we train the Schrodinger Audio-Visual Editor (SAVE), an end-to-end flow-matching model that edits audio and video in parallel while keeping them aligned throughout processing. SAVE incorporates a Schrodinger Bridge that learns a direct transport from source to target audiovisual mixtures. Our evaluation demonstrates that the proposed SAVE model is able to remove the target objects in audio and visual content while preserving the remaining content, with stronger temporal synchronization and audiovisual semantic correspondence compared with pairwise combinations of an audio editor and a video editor.