Video Mask Transfiner for High-Quality Video Instance Segmentation

TL;DR

This paper introduces the Video Mask Transfiner (VMT), a transformer-based method for high-quality, temporally consistent video instance segmentation, along with a new dataset for benchmarking detailed masks.

Contribution

The paper presents a novel VMT architecture leveraging high-resolution features and a new annotation refinement approach, along with the HQ-YTVIS dataset for better benchmarking.

Findings

VMT achieves more detailed and stable masks.

Automated annotation refinement improves training data quality.

VMT outperforms recent methods on multiple benchmarks.

Abstract

While Video Instance Segmentation (VIS) has seen rapid progress, current approaches struggle to predict high-quality masks with accurate boundary details. Moreover, the predicted segmentations often fluctuate over time, suggesting that temporal consistency cues are neglected or not fully utilized. In this paper, we set out to tackle these issues, with the aim of achieving highly detailed and more temporally stable mask predictions for VIS. We first propose the Video Mask Transfiner (VMT) method, capable of leveraging fine-grained high-resolution features thanks to a highly efficient video transformer structure. Our VMT detects and groups sparse error-prone spatio-temporal regions of each tracklet in the video segment, which are then refined using both local and instance-level cues. Second, we identify that the coarse boundary annotations of the popular YouTube-VIS dataset constitute a major limiting factor. Based on our VMT architecture, we therefore design an automated annotation refinement approach by iterative training and self-correction. To benchmark high-quality mask predictions for VIS, we introduce the HQ-YTVIS dataset, consisting of a manually re-annotated test set and our automatically refined training data. We compare VMT with the most recent state-of-the-art methods on the HQ-YTVIS, as well as the Youtube-VIS, OVIS and BDD100K MOTS benchmarks. Experimental results clearly demonstrate the efficacy and effectiveness of our method on segmenting complex and dynamic objects, by capturing precise details.