Class-relevant Patch Embedding Selection for Few-Shot Image Classification

TL;DR

This paper introduces a method for selecting class-relevant patch embeddings in few-shot image classification, improving pattern recognition by filtering out irrelevant patches and enhancing model performance.

Contribution

The proposed approach uses class embeddings to filter and select top patch embeddings, effectively improving few-shot classification accuracy with a simple and efficient method.

Findings

Outperforms state-of-the-art baselines in 5-shot and 1-shot tasks

Enhances pattern recognition by filtering irrelevant patches

Proves effective and computationally efficient

Abstract

Effective image classification hinges on discerning relevant features from both foreground and background elements, with the foreground typically holding the critical information. While humans adeptly classify images with limited exposure, artificial neural networks often struggle with feature selection from rare samples. To address this challenge, we propose a novel method for selecting class-relevant patch embeddings. Our approach involves splitting support and query images into patches, encoding them using a pre-trained Vision Transformer (ViT) to obtain class embeddings and patch embeddings, respectively. Subsequently, we filter patch embeddings using class embeddings to retain only the class-relevant ones. For each image, we calculate the similarity between class embedding and each patch embedding, sort the similarity sequence in descending order, and only retain top-ranked patch embeddings. By prioritizing similarity between the class embedding and patch embeddings, we select top-ranked patch embeddings to be fused with class embedding to form a comprehensive image representation, enhancing pattern recognition across instances. Our strategy effectively mitigates the impact of class-irrelevant patch embeddings, yielding improved performance in pre-trained models. Extensive experiments on popular few-shot classification benchmarks demonstrate the simplicity, efficacy, and computational efficiency of our approach, outperforming state-of-the-art baselines under both 5-shot and 1-shot scenarios.