SyncVIS: Synchronized Video Instance Segmentation

TL;DR

SyncVIS introduces a synchronized video instance segmentation framework that explicitly models video-level and frame-level queries, improving performance on challenging benchmarks by promoting mutual learning and easier optimization.

Contribution

The paper proposes a novel synchronized modeling framework for VIS that explicitly incorporates video-level queries and synchronization modules, addressing limitations of asynchronous designs.

Findings

Achieves state-of-the-art results on YouTube-VIS and OVIS benchmarks.

Demonstrates the effectiveness of synchronized query modeling.

Validates generality across multiple challenging datasets.

Abstract

Recent DETR-based methods have advanced the development of Video Instance Segmentation (VIS) through transformers' efficiency and capability in modeling spatial and temporal information. Despite harvesting remarkable progress, existing works follow asynchronous designs, which model video sequences via either video-level queries only or adopting query-sensitive cascade structures, resulting in difficulties when handling complex and challenging video scenarios. In this work, we analyze the cause of this phenomenon and the limitations of the current solutions, and propose to conduct synchronized modeling via a new framework named SyncVIS. Specifically, SyncVIS explicitly introduces video-level query embeddings and designs two key modules to synchronize video-level query with frame-level query embeddings: a synchronized video-frame modeling paradigm and a synchronized embedding optimization strategy. The former attempts to promote the mutual learning of frame- and video-level embeddings with each other and the latter divides large video sequences into small clips for easier optimization. Extensive experimental evaluations are conducted on the challenging YouTube-VIS 2019 & 2021 & 2022, and OVIS benchmarks and SyncVIS achieves state-of-the-art results, which demonstrates the effectiveness and generality of the proposed approach. The code is available at https://github.com/rkzheng99/SyncVIS.