Unified Embedding Alignment for Open-Vocabulary Video Instance Segmentation

TL;DR

This paper introduces OVFormer, a novel approach for open-vocabulary video instance segmentation that aligns embeddings and leverages temporal consistency, significantly improving performance and generalization over previous methods.

Contribution

OVFormer is a new baseline that uses unified embedding alignment and video-based training to enhance open-vocabulary VIS performance and generalization.

Findings

Achieves 21.9 mAP on LV-VIS with ResNet-50, surpassing previous state-of-the-art by 7.7.

Demonstrates strong zero-shot generalization on YouTube-VIS 2019 and OVIS datasets.

Utilizes a lightweight module for embedding alignment and semi-online inference for temporal consistency.

Abstract

Open-Vocabulary Video Instance Segmentation (VIS) is attracting increasing attention due to its ability to segment and track arbitrary objects. However, the recent Open-Vocabulary VIS attempts obtained unsatisfactory results, especially in terms of generalization ability of novel categories. We discover that the domain gap between the VLM features (e.g., CLIP) and the instance queries and the underutilization of temporal consistency are two central causes. To mitigate these issues, we design and train a novel Open-Vocabulary VIS baseline called OVFormer. OVFormer utilizes a lightweight module for unified embedding alignment between query embeddings and CLIP image embeddings to remedy the domain gap. Unlike previous image-based training methods, we conduct video-based model training and deploy a semi-online inference scheme to fully mine the temporal consistency in the video. Without bells and whistles, OVFormer achieves 21.9 mAP with a ResNet-50 backbone on LV-VIS, exceeding the previous state-of-the-art performance by 7.7. Extensive experiments on some Close-Vocabulary VIS datasets also demonstrate the strong zero-shot generalization ability of OVFormer (+ 7.6 mAP on YouTube-VIS 2019, + 3.9 mAP on OVIS). Code is available at https://github.com/fanghaook/OVFormer.