Video Instance Segmentation with a Propose-Reduce Paradigm

TL;DR

This paper introduces the Propose-Reduce paradigm for video instance segmentation, enabling complete sequence generation in a single step and improving robustness and recall, leading to state-of-the-art results.

Contribution

The paper proposes a novel Propose-Reduce paradigm and a sequence propagation head for improved long-term video instance segmentation.

Findings

Achieved 47.6% AP on YouTube-VIS dataset.

Achieved 70.4% J&F on DAVIS-UVOS dataset.

Outperforms previous methods with state-of-the-art results.

Abstract

Video instance segmentation (VIS) aims to segment and associate all instances of predefined classes for each frame in videos. Prior methods usually obtain segmentation for a frame or clip first, and merge the incomplete results by tracking or matching. These methods may cause error accumulation in the merging step. Contrarily, we propose a new paradigm -- Propose-Reduce, to generate complete sequences for input videos by a single step. We further build a sequence propagation head on the existing image-level instance segmentation network for long-term propagation. To ensure robustness and high recall of our proposed framework, multiple sequences are proposed where redundant sequences of the same instance are reduced. We achieve state-of-the-art performance on two representative benchmark datasets -- we obtain 47.6% in terms of AP on YouTube-VIS validation set and 70.4% for J&F on DAVIS-UVOS validation set. Code is available at https://github.com/dvlab-research/ProposeReduce.