PMCE: Probabilistic Multi-Granularity Semantics with Caption-Guided Enhancement for Few-Shot Learning

TL;DR

PMCE introduces a probabilistic framework for few-shot learning that combines multi-granularity semantics, caption-guided enhancement, and a knowledge bank to improve classification accuracy with limited data.

Contribution

It proposes a novel PMCE framework that leverages class name embeddings, caption-based instance descriptions, and a knowledge bank for improved few-shot learning.

Findings

Achieves up to 7.71% accuracy gain on MiniImageNet 1-shot.

Consistently outperforms strong semantic baselines across four benchmarks.

Utilizes caption-guided enhancement to stabilize noisy descriptions.

Abstract

Few-shot learning aims to identify novel categories from only a handful of labeled samples, where prototypes estimated from scarce data are often biased and generalize poorly. Semantic-based methods alleviate this by introducing coarse class-level information, but they are mostly applied on the support side, leaving query representations unchanged. In this paper, we present PMCE, a Probabilistic few-shot framework that leverages Multi-granularity semantics with Caption-guided Enhancement. PMCE constructs a nonparametric knowledge bank that stores visual statistics for each category as well as CLIP-encoded class name embeddings of the base classes. At meta-test time, the most relevant base classes are retrieved based on the similarities of class name embeddings for each novel category. These statistics are then aggregated into category-specific prior information and fused with the support set prototypes via a simple MAP update. Simultaneously, a frozen BLIP captioner provides label-free instance-level image descriptions, and a lightweight enhancer trained on base classes optimizes both support prototypes and query features under an inductive protocol with a consistency regularization to stabilize noisy captions. Experiments on four benchmarks show that PMCE consistently improves over strong baselines, achieving up to 7.71% absolute gain over the strongest semantic competitor on MiniImageNet in the 1-shot setting. Our code is available at https://anonymous.4open.science/r/PMCE-275D