No time to train! Training-Free Reference-Based Instance Segmentation

TL;DR

This paper introduces a training-free, reference-based instance segmentation method that leverages foundation models' semantic priors to identify object regions across images, reducing annotation and prompt engineering efforts.

Contribution

It proposes a novel multi-stage, training-free approach utilizing memory banks, representation aggregation, and semantic-aware matching for segmentation based on reference images.

Findings

Achieves state-of-the-art results on COCO FSOD with 36.8% nAP.

Outperforms existing training-free methods on Cross-Domain FSOD with 22.4% nAP.

Significantly improves segmentation metrics across benchmarks.

Abstract

The performance of image segmentation models has historically been constrained by the high cost of collecting large-scale annotated data. The Segment Anything Model (SAM) alleviates this original problem through a promptable, semantics-agnostic, segmentation paradigm and yet still requires manual visual-prompts or complex domain-dependent prompt-generation rules to process a new image. Towards reducing this new burden, our work investigates the task of object segmentation when provided with, alternatively, only a small set of reference images. Our key insight is to leverage strong semantic priors, as learned by foundation models, to identify corresponding regions between a reference and a target image. We find that correspondences enable automatic generation of instance-level segmentation masks for downstream tasks and instantiate our ideas via a multi-stage, training-free method incorporating (1) memory bank construction; (2) representation aggregation and (3) semantic-aware feature matching. Our experiments show significant improvements on segmentation metrics, leading to state-of-the-art performance on COCO FSOD (36.8% nAP), PASCAL VOC Few-Shot (71.2% nAP50) and outperforming existing training-free approaches on the Cross-Domain FSOD benchmark (22.4% nAP).