Few-Shot Learning with Visual Distribution Calibration and Cross-Modal Distribution Alignment

TL;DR

This paper introduces a novel few-shot learning method that calibrates visual feature distributions and aligns them with language features using a selective attack module, cross-modal distribution alignment, and data augmentation, leading to improved performance.

Contribution

The paper proposes a new approach combining visual distribution calibration and cross-modal alignment with a selective attack module and prototype-based optimization, advancing few-shot vision-language learning.

Findings

Outperforms prior methods on 11 datasets

Effective visual feature calibration via selective attack

Improved alignment between visual and language features

Abstract

Pre-trained vision-language models have inspired much research on few-shot learning. However, with only a few training images, there exist two crucial problems: (1) the visual feature distributions are easily distracted by class-irrelevant information in images, and (2) the alignment between the visual and language feature distributions is difficult. To deal with the distraction problem, we propose a Selective Attack module, which consists of trainable adapters that generate spatial attention maps of images to guide the attacks on class-irrelevant image areas. By messing up these areas, the critical features are captured and the visual distributions of image features are calibrated. To better align the visual and language feature distributions that describe the same object class, we propose a cross-modal distribution alignment module, in which we introduce a vision-language prototype for each class to align the distributions, and adopt the Earth Mover's Distance (EMD) to optimize the prototypes. For efficient computation, the upper bound of EMD is derived. In addition, we propose an augmentation strategy to increase the diversity of the images and the text prompts, which can reduce overfitting to the few-shot training images. Extensive experiments on 11 datasets demonstrate that our method consistently outperforms prior arts in few-shot learning. The implementation code will be available at https://github.com/bhrqw/SADA.