TAVP: Task-Adaptive Visual Prompt for Cross-domain Few-shot Segmentation

TL;DR

This paper introduces TAVP, a novel task-adaptive visual prompt framework that enhances cross-domain few-shot segmentation by leveraging multi-level feature fusion and a class-domain agnostic prompt generation, significantly improving accuracy over previous methods.

Contribution

The paper proposes a new auto-visual prompt framework with a generative approach and specialized prototypes for better knowledge transfer in cross-domain few-shot segmentation.

Findings

Outperforms state-of-the-art in four cross-domain datasets.

Achieves 1.3% accuracy improvement in 1-shot setting.

Achieves 11.76% accuracy improvement in 5-shot setting.

Abstract

While large visual models (LVM) demonstrated significant potential in image understanding, due to the application of large-scale pre-training, the Segment Anything Model (SAM) has also achieved great success in the field of image segmentation, supporting flexible interactive cues and strong learning capabilities. However, SAM's performance often falls short in cross-domain and few-shot applications. Previous work has performed poorly in transferring prior knowledge from base models to new applications. To tackle this issue, we propose a task-adaptive auto-visual prompt framework, a new paradigm for Cross-dominan Few-shot segmentation (CD-FSS). First, a Multi-level Feature Fusion (MFF) was used for integrated feature extraction as prior knowledge. Besides, we incorporate a Class Domain Task-Adaptive Auto-Prompt (CDTAP) module to enable class-domain agnostic feature extraction and generate high-quality, learnable visual prompts. This significant advancement uses a unique generative approach to prompts alongside a comprehensive model structure and specialized prototype computation. While ensuring that the prior knowledge of SAM is not discarded, the new branch disentangles category and domain information through prototypes, guiding it in adapting the CD-FSS. Comprehensive experiments across four cross-domain datasets demonstrate that our model outperforms the state-of-the-art CD-FSS approach, achieving an average accuracy improvement of 1.3\% in the 1-shot setting and 11.76\% in the 5-shot setting.