VITA: Video Instance Segmentation via Object Token Association

TL;DR

VITA introduces a novel object token association method for offline Video Instance Segmentation, leveraging object-oriented information and transformer-based models to achieve state-of-the-art results efficiently.

Contribution

The paper presents VITA, a new framework that uses object tokens and a transformer-based approach for video instance segmentation without relying on spatio-temporal backbone features.

Findings

Achieves state-of-the-art AP scores on VIS benchmarks.

Handles long, high-resolution videos efficiently.

Operates with a frozen image detector, simplifying training.

Abstract

We introduce a novel paradigm for offline Video Instance Segmentation (VIS), based on the hypothesis that explicit object-oriented information can be a strong clue for understanding the context of the entire sequence. To this end, we propose VITA, a simple structure built on top of an off-the-shelf Transformer-based image instance segmentation model. Specifically, we use an image object detector as a means of distilling object-specific contexts into object tokens. VITA accomplishes video-level understanding by associating frame-level object tokens without using spatio-temporal backbone features. By effectively building relationships between objects using the condensed information, VITA achieves the state-of-the-art on VIS benchmarks with a ResNet-50 backbone: 49.8 AP, 45.7 AP on YouTube-VIS 2019 & 2021, and 19.6 AP on OVIS. Moreover, thanks to its object token-based structure that is disjoint from the backbone features, VITA shows several practical advantages that previous offline VIS methods have not explored - handling long and high-resolution videos with a common GPU, and freezing a frame-level detector trained on image domain. Code is available at https://github.com/sukjunhwang/VITA.