Temporal Transductive Inference for Few-Shot Video Object Segmentation

TL;DR

This paper introduces a temporal transductive inference method for few-shot video object segmentation that leverages temporal consistency and regularization to improve segmentation accuracy and robustness.

Contribution

The paper proposes a novel TTI approach with global and local temporal constraints, enhancing temporal coherence and reducing overfitting in FS-VOS.

Findings

Outperforms state-of-the-art meta-learning methods by 2.8% in mean IoU on YouTube-VIS.

Introduces new exhaustive benchmarks and realistic evaluation paradigms.

Demonstrates the effectiveness of spatiotemporal regularizers in improving segmentation quality.

Abstract

Few-shot video object segmentation (FS-VOS) aims at segmenting video frames using a few labelled examples of classes not seen during initial training. In this paper, we present a simple but effective temporal transductive inference (TTI) approach that leverages temporal consistency in the unlabelled video frames during few-shot inference. Key to our approach is the use of both global and local temporal constraints. The objective of the global constraint is to learn consistent linear classifiers for novel classes across the image sequence, whereas the local constraint enforces the proportion of foreground/background regions in each frame to be coherent across a local temporal window. These constraints act as spatiotemporal regularizers during the transductive inference to increase temporal coherence and reduce overfitting on the few-shot support set. Empirically, our model outperforms state-of-the-art meta-learning approaches in terms of mean intersection over union on YouTube-VIS by 2.8%. In addition, we introduce improved benchmarks that are exhaustively labelled (i.e. all object occurrences are labelled, unlike the currently available), and present a more realistic evaluation paradigm that targets data distribution shift between training and testing sets. Our empirical results and in-depth analysis confirm the added benefits of the proposed spatiotemporal regularizers to improve temporal coherence and overcome certain overfitting scenarios.