Beyond Mask: Rethinking Guidance Types in Few-shot Segmentation

TL;DR

This paper introduces UniFSS, a universal vision-language framework for few-shot segmentation that integrates multiple guidance types, significantly improving performance and surpassing state-of-the-art methods, including weakly annotated paradigms.

Contribution

The work generalizes guidance types in FSS by proposing seven paradigms and develops UniFSS, a framework leveraging vision-language models for improved segmentation performance.

Findings

UniFSS outperforms existing state-of-the-art methods.

Weakly annotated box guidance surpasses mask guidance.

High-level spatial correction enhances semantic understanding.

Abstract

Existing few-shot segmentation (FSS) methods mainly focus on prototype feature generation and the query-support matching mechanism. As a crucial prompt for generating prototype features, the pair of image-mask types in the support set has become the default setting. However, various types such as image, text, box, and mask all can provide valuable information regarding the objects in context, class, localization, and shape appearance. Existing work focuses on specific combinations of guidance, leading FSS into different research branches. Rethinking guidance types in FSS is expected to explore the efficient joint representation of the coupling between the support set and query set, giving rise to research trends in the weakly or strongly annotated guidance to meet the customized requirements of practical users. In this work, we provide the generalized FSS with seven guidance paradigms and develop a universal vision-language framework (UniFSS) to integrate prompts from text, mask, box, and image. Leveraging the advantages of large-scale pre-training vision-language models in textual and visual embeddings, UniFSS proposes high-level spatial correction and embedding interactive units to overcome the semantic ambiguity drawbacks typically encountered by pure visual matching methods when facing intra-class appearance diversities. Extensive experiments show that UniFSS significantly outperforms the state-of-the-art methods. Notably, the weakly annotated class-aware box paradigm even surpasses the finely annotated mask paradigm.