APSeg: Auto-Prompt Network for Cross-Domain Few-Shot Semantic Segmentation

TL;DR

APSeg introduces an auto-prompt network leveraging a dual prototype transformation and a meta prompt generator to enhance cross-domain few-shot semantic segmentation without fine-tuning, outperforming existing methods.

Contribution

The paper proposes APSeg, a novel auto-prompt network with dual prototype transformation and meta prompt generation for improved cross-domain FSS.

Findings

Outperforms state-of-the-art in cross-domain FSS by 5.24% and 3.10%.

Effective domain-agnostic feature transformation via DPAT.

Automatic prompt generation eliminates manual intervention.

Abstract

Few-shot semantic segmentation (FSS) endeavors to segment unseen classes with only a few labeled samples. Current FSS methods are commonly built on the assumption that their training and application scenarios share similar domains, and their performances degrade significantly while applied to a distinct domain. To this end, we propose to leverage the cutting-edge foundation model, the Segment Anything Model (SAM), for generalization enhancement. The SAM however performs unsatisfactorily on domains that are distinct from its training data, which primarily comprise natural scene images, and it does not support automatic segmentation of specific semantics due to its interactive prompting mechanism. In our work, we introduce APSeg, a novel auto-prompt network for cross-domain few-shot semantic segmentation (CD-FSS), which is designed to be auto-prompted for guiding cross-domain segmentation. Specifically, we propose a Dual Prototype Anchor Transformation (DPAT) module that fuses pseudo query prototypes extracted based on cycle-consistency with support prototypes, allowing features to be transformed into a more stable domain-agnostic space. Additionally, a Meta Prompt Generator (MPG) module is introduced to automatically generate prompt embeddings, eliminating the need for manual visual prompts. We build an efficient model which can be applied directly to target domains without fine-tuning. Extensive experiments on four cross-domain datasets show that our model outperforms the state-of-the-art CD-FSS method by 5.24% and 3.10% in average accuracy on 1-shot and 5-shot settings, respectively.