Multiscale Memory Comparator Transformer for Few-Shot Video Segmentation

TL;DR

This paper introduces a multiscale memory transformer for few-shot video segmentation that preserves detailed features across scales, leading to improved accuracy and state-of-the-art results.

Contribution

It proposes a novel multiscale memory comparator within a transformer decoder that maintains detailed features across scales, enhancing few-shot video segmentation performance.

Findings

Outperforms baseline methods on few-shot video object segmentation

Achieves state-of-the-art results across multiple tasks

Provides empirical insights into multiscale information exchange

Abstract

Few-shot video segmentation is the task of delineating a specific novel class in a query video using few labelled support images. Typical approaches compare support and query features while limiting comparisons to a single feature layer and thereby ignore potentially valuable information. We present a meta-learned Multiscale Memory Comparator (MMC) for few-shot video segmentation that combines information across scales within a transformer decoder. Typical multiscale transformer decoders for segmentation tasks learn a compressed representation, their queries, through information exchange across scales. Unlike previous work, we instead preserve the detailed feature maps during across scale information exchange via a multiscale memory transformer decoding to reduce confusion between the background and novel class. Integral to the approach, we investigate multiple forms of information exchange across scales in different tasks and provide insights with empirical evidence on which to use in each task. The overall comparisons among query and support features benefit from both rich semantics and precise localization. We demonstrate our approach primarily on few-shot video object segmentation and an adapted version on the fully supervised counterpart. In all cases, our approach outperforms the baseline and yields state-of-the-art performance. Our code is publicly available at https://github.com/MSiam/MMC-MultiscaleMemory.