SAMIC: Segment Anything with In-Context Spatial Prompt Engineering

TL;DR

SAMIC is a compact network that leverages vision foundation models to efficiently perform few-shot segmentation across various domains, significantly reducing model size and training data requirements while maintaining competitive performance.

Contribution

The paper introduces SAMIC, a small network that prompts vision foundation models for few-shot segmentation, enabling domain-specific applications with minimal training data and model size.

Findings

SAMIC achieves state-of-the-art results on multiple segmentation benchmarks.

SAMIC is 94% smaller than leading models with comparable performance.

It performs well even with only 1/5th of the training data.

Abstract

Few-shot segmentation is the problem of learning to identify specific types of objects (e.g., airplanes) in images from a small set of labeled reference images. The current state of the art is driven by resource-intensive construction of models for every new domain-specific application. Such models must be trained on enormous labeled datasets of unrelated objects (e.g., cars, trains, animals) so that their ``knowledge'' can be transferred to new types of objects. In this paper, we show how to leverage existing vision foundation models (VFMs) to reduce the incremental cost of creating few-shot segmentation models for new domains. Specifically, we introduce SAMIC, a small network that learns how to prompt VFMs in order to segment new types of objects in domain-specific applications. SAMIC enables any task to be approached as a few-shot learning problem. At 2.6 million parameters, it is 94% smaller than the leading models (e.g., having ResNet 101 backbone with 45+ million parameters). Even using 1/5th of the training data provided by one-shot benchmarks, SAMIC is competitive with, or sets the state of the art, on a variety of few-shot and semantic segmentation datasets including COCO-$20^i$, Pascal-$5^i$, PerSeg, FSS-1000, and NWPU VHR-10.