Instance As Identity: A Generic Online Paradigm for Video Instance Segmentation

TL;DR

The paper introduces Instance As Identity (IAI), a novel online video instance segmentation paradigm that models temporal information efficiently for detection and tracking, achieving state-of-the-art results on multiple benchmarks.

Contribution

It proposes a new online VIS framework with an identification module and an association module, compatible with various image models, advancing real-time video segmentation.

Findings

Outperforms all online competitors on YouTube-VIS benchmarks.

Achieves state-of-the-art performance on the challenging OVIS dataset.

Demonstrates flexibility by integrating with different image models.

Abstract

Modeling temporal information for both detection and tracking in a unified framework has been proved a promising solution to video instance segmentation (VIS). However, how to effectively incorporate the temporal information into an online model remains an open problem. In this work, we propose a new online VIS paradigm named Instance As Identity (IAI), which models temporal information for both detection and tracking in an efficient way. In detail, IAI employs a novel identification module to predict identification number for tracking instances explicitly. For passing temporal information cross frame, IAI utilizes an association module which combines current features and past embeddings. Notably, IAI can be integrated with different image models. We conduct extensive experiments on three VIS benchmarks. IAI outperforms all the online competitors on YouTube-VIS-2019 (ResNet-101 43.7 mAP) and YouTube-VIS-2021 (ResNet-50 38.0 mAP). Surprisingly, on the more challenging OVIS, IAI achieves SOTA performance (20.6 mAP). Code is available at https://github.com/zfonemore/IAI