High-Performance Few-Shot Segmentation with Foundation Models: An Empirical Study

TL;DR

This paper investigates how foundation models can enhance few-shot segmentation by extracting implicit knowledge, demonstrating significant performance improvements over traditional pre-trained models through extensive experiments.

Contribution

It introduces a novel FSS framework leveraging foundation models' implicit knowledge, outperforming existing methods and systematically analyzing various foundation models' effectiveness.

Findings

Foundation models provide more beneficial implicit knowledge for FSS than classification pre-trained models.

The proposed approach achieves a 17.5% improvement on COCO-20i over previous state-of-the-art methods.

Extensive experiments validate the effectiveness of leveraging foundation models in FSS.

Abstract

Existing few-shot segmentation (FSS) methods mainly focus on designing novel support-query matching and self-matching mechanisms to exploit implicit knowledge in pre-trained backbones. However, the performance of these methods is often constrained by models pre-trained on classification tasks. The exploration of what types of pre-trained models can provide more beneficial implicit knowledge for FSS remains limited. In this paper, inspired by the representation consistency of foundational computer vision models, we develop a FSS framework based on foundation models. To be specific, we propose a simple approach to extract implicit knowledge from foundation models to construct coarse correspondence and introduce a lightweight decoder to refine coarse correspondence for fine-grained segmentation. We systematically summarize the performance of various foundation models on FSS and discover that the implicit knowledge within some of these models is more beneficial for FSS than models pre-trained on classification tasks. Extensive experiments on two widely used datasets demonstrate the effectiveness of our approach in leveraging the implicit knowledge of foundation models. Notably, the combination of DINOv2 and DFN exceeds previous state-of-the-art methods by 17.5% on COCO-20i. Code is available at https://github.com/DUT-CSJ/FoundationFSS.