Robust Promptable Video Object Segmentation

TL;DR

This paper introduces a new benchmark and a robust video object segmentation method called MoGA, which significantly improves performance under real-world corruptions by handling object-specific degradation and maintaining temporal consistency.

Contribution

It provides the first comprehensive benchmark for robust PVOS and proposes MoGA, a novel method that leverages object memory to enhance robustness against diverse corruptions.

Findings

MoGA outperforms existing methods across various corruption types.

The benchmark includes 351 videos with over 2,500 object masks under adverse conditions.

Synthetic training data with diverse corruptions improves model robustness.

Abstract

The performance of promptable video object segmentation (PVOS) models substantially degrades under input corruptions, which prevents PVOS deployment in safety-critical domains. This paper offers the first comprehensive study on robust PVOS (RobustPVOS). We first construct a new, comprehensive benchmark with two real-world evaluation datasets of 351 video clips and more than 2,500 object masks under real-world adverse conditions. At the same time, we generate synthetic training data by applying diverse and temporally varying corruptions to existing VOS datasets. Moreover, we present a new RobustPVOS method, dubbed Memory-object-conditioned Gated-rank Adaptation (MoGA). The key to successfully performing RobustPVOS is two-fold: effectively handling object-specific degradation and ensuring temporal consistency in predictions. MoGA leverages object-specific representations maintained in memory across frames to condition the robustification process, which allows the model to handle each tracked object differently in a temporally consistent way. Extensive experiments on our benchmark validate MoGA's efficacy, showing consistent and significant improvements across diverse corruption types on both synthetic and real-world datasets, establishing a strong baseline for future RobustPVOS research. Our benchmark is publicly available at https://sohyun-l.github.io/RobustPVOS_project_page/.