Beyond Visual Cues: Leveraging General Semantics as Support for Few-Shot Segmentation

TL;DR

This paper introduces a novel language-driven approach for few-shot segmentation that leverages large language models and multi-modal alignment to provide unbiased, detailed attribute guidance, outperforming existing methods.

Contribution

The paper proposes LDAG, a new architecture that uses language descriptions and multi-modal matching to improve few-shot segmentation, addressing intra-class variation issues.

Findings

Outperforms existing few-shot segmentation methods.

Achieves state-of-the-art performance on benchmark datasets.

Demonstrates the effectiveness of language-driven attribute guidance.

Abstract

Few-shot segmentation (FSS) aims to segment novel classes under the guidance of limited support samples by a meta-learning paradigm. Existing methods mainly mine references from support images as meta guidance. However, due to intra-class variations among visual representations, the meta information extracted from support images cannot produce accurate guidance to segment untrained classes. In this paper, we argue that the references from support images may not be essential, the key to the support role is to provide unbiased meta guidance for both trained and untrained classes. We then introduce a Language-Driven Attribute Generalization (LDAG) architecture to utilize inherent target property language descriptions to build robust support strategy. Specifically, to obtain an unbiased support representation, we design a Multi-attribute Enhancement (MaE) module, which produces multiple detailed attribute descriptions of the target class through Large Language Models (LLMs), and then builds refined visual-text prior guidance utilizing multi-modal matching. Meanwhile, due to text-vision modal shift, attribute text struggles to promote visual feature representation, we design a Multi-modal Attribute Alignment (MaA) to achieve cross-modal interaction between attribute texts and visual feature. Experiments show that our proposed method outperforms existing approaches by a clear margin and achieves the new state-of-the art performance. The code will be released.