Object Propagation via Inter-Frame Attentions for Temporally Stable Video Instance Segmentation

TL;DR

This paper introduces a novel video instance segmentation method that uses inter-frame attention to improve temporal stability and detection consistency, outperforming previous methods on the YouTube-VIS benchmark.

Contribution

The proposed approach leverages inter-frame attentions to recover missing object detections, enhancing temporal consistency in video segmentation.

Findings

Achieves 36.0% mAP on YouTube-VIS benchmark

Outperforms previous state-of-the-art methods

Operates fully online without future frame access

Abstract

Video instance segmentation aims to detect, segment, and track objects in a video. Current approaches extend image-level segmentation algorithms to the temporal domain. However, this results in temporally inconsistent masks. In this work, we identify the mask quality due to temporal stability as a performance bottleneck. Motivated by this, we propose a video instance segmentation method that alleviates the problem due to missing detections. Since this cannot be solved simply using spatial information, we leverage temporal context using inter-frame attentions. This allows our network to refocus on missing objects using box predictions from the neighbouring frame, thereby overcoming missing detections. Our method significantly outperforms previous state-of-the-art algorithms using the Mask R-CNN backbone, by achieving 36.0% mAP on the YouTube-VIS benchmark. Additionally, our method is completely online and requires no future frames. Our code is publicly available at https://github.com/anirudh-chakravarthy/ObjProp.