From Ideal to Real: Stable Video Object Removal under Imperfect Conditions

TL;DR

This paper introduces SVOR, a robust video object removal framework that effectively handles shadows, motion, and mask defects, achieving state-of-the-art results in real-world scenarios.

Contribution

The paper proposes three novel techniques—MUSE, DA-Seg, and a two-stage training process—that significantly improve the stability and robustness of video object removal under imperfect conditions.

Findings

SVOR outperforms existing methods on multiple datasets.

The framework effectively removes shadows and reflections.

It maintains temporal stability and visual consistency in challenging scenarios.

Abstract

Removing objects from videos remains difficult in the presence of real-world imperfections such as shadows, abrupt motion, and defective masks. Existing diffusion-based video inpainting models often struggle to maintain temporal stability and visual consistency under these challenges. We propose Stable Video Object Removal (SVOR), a robust framework that achieves shadow-free, flicker-free, and mask-defect-tolerant removal through three key designs: (1) Mask Union for Stable Erasure (MUSE), a windowed union strategy applied during temporal mask downsampling to preserve all target regions observed within each window, effectively handling abrupt motion and reducing missed removals; (2) Denoising-Aware Segmentation (DA-Seg), a lightweight segmentation head on a decoupled side branch equipped with Denoising-Aware AdaLN and trained with mask degradation to provide an internal diffusion-aware localization prior without affecting content generation; and (3) Curriculum Two-Stage Training: where Stage I performs self-supervised pretraining on unpaired real-background videos with online random masks to learn realistic background and temporal priors, and Stage II refines on synthetic pairs using mask degradation and side-effect-weighted losses, jointly removing objects and their associated shadows/reflections while improving cross-domain robustness. Extensive experiments show that SVOR attains new state-of-the-art results across multiple datasets and degraded-mask benchmarks, advancing video object removal from ideal settings toward real-world applications. Project page: https://xiaomi-research.github.io/svor/.