Offline-to-Online Knowledge Distillation for Video Instance Segmentation

TL;DR

This paper introduces a novel offline-to-online knowledge distillation approach for video instance segmentation, enhancing model robustness and achieving state-of-the-art results on multiple challenging datasets.

Contribution

It proposes a new knowledge distillation framework with query filtering and association, improving online model performance by leveraging offline model knowledge.

Findings

Significant performance improvements on YTVIS-21, YTVIS-22, and OVIS datasets.

Effective knowledge transfer through query filtering and data augmentation.

Achieved state-of-the-art mAP scores in video instance segmentation.

Abstract

In this paper, we present offline-to-online knowledge distillation (OOKD) for video instance segmentation (VIS), which transfers a wealth of video knowledge from an offline model to an online model for consistent prediction. Unlike previous methods that having adopting either an online or offline model, our single online model takes advantage of both models by distilling offline knowledge. To transfer knowledge correctly, we propose query filtering and association (QFA), which filters irrelevant queries to exact instances. Our KD with QFA increases the robustness of feature matching by encoding object-centric features from a single frame supplemented by long-range global information. We also propose a simple data augmentation scheme for knowledge distillation in the VIS task that fairly transfers the knowledge of all classes into the online model. Extensive experiments show that our method significantly improves the performance in video instance segmentation, especially for challenging datasets including long, dynamic sequences. Our method also achieves state-of-the-art performance on YTVIS-21, YTVIS-22, and OVIS datasets, with mAP scores of 46.1%, 43.6%, and 31.1%, respectively.