Inter- and Intra-image Refinement for Few Shot Segmentation

TL;DR

This paper introduces the IIR model for few shot segmentation, which refines support-query features through inter- and intra-image techniques, significantly improving accuracy across multiple benchmarks.

Contribution

The paper proposes a novel IIR model that addresses inter- and intra-image discrepancies in FSS, utilizing dual prototypes and directional dropout for enhanced segmentation performance.

Findings

Achieves state-of-the-art results on 9 benchmarks.

Effectively reduces intra-class gap and inter-class interference.

Demonstrates robustness across standard, part, and cross-domain FSS.

Abstract

Deep neural networks for semantic segmentation rely on large-scale annotated datasets, leading to an annotation bottleneck that motivates few shot semantic segmentation (FSS) which aims to generalize to novel classes with minimal labeled exemplars. Most existing FSS methods adopt a prototype-based paradigm, which generates query prior map by extracting masked-area features from support images and then makes predictions guided by the prior map. However, they suffer from two critical limitations induced by inter- and intra-image discrepancies: 1) The intra-class gap between support and query images, caused by single-prototype representation, results in scattered and noisy prior maps; 2) The inter-class interference from visually similar but semantically distinct regions leads to inconsistent support-query feature matching and erroneous predictions. To address these issues, we propose the Inter- and Intra-image Refinement (IIR) model. The model contains an inter-image class activation mapping based method that generates two prototypes for class-consistent region matching, including core discriminative features and local specific features, and yields an accurate and robust prior map. For intra-image refinement, a directional dropout mechanism is introduced to mask inconsistent support-query feature pairs in cross attention, thereby enhancing decoder performance. Extensive experiments demonstrate that IIR achieves state-of-the-art performance on 9 benchmarks, covering standard FSS, part FSS, and cross-domain FSS. Our source code is available at \href{https://github.com/forypipi/IIR}{https://github.com/forypipi/IIR}.