Mind the Gap Between Prototypes and Images in Cross-domain Finetuning

TL;DR

This paper identifies a modality gap between prototypes and images in cross-domain few-shot classification and proposes CoPA, a method that adapts different transformations for each, leading to improved performance and representation clustering.

Contribution

The paper introduces CoPA, a novel contrastive adaptation method that independently transforms prototypes and images, addressing the modality gap in cross-domain few-shot learning.

Findings

CoPA achieves state-of-the-art results on Meta-Dataset.

It learns better representation clusters and enlarges the prototype-image gap.

CoPA minimizes validation loss at an enlarged gap.

Abstract

In cross-domain few-shot classification (CFC), recent works mainly focus on adapting a simple transformation head on top of a frozen pre-trained backbone with few labeled data to project embeddings into a task-specific metric space where classification can be performed by measuring similarities between image instance and prototype representations. Technically, an assumption implicitly adopted in such a framework is that the prototype and image instance embeddings share the same representation transformation. However, in this paper, we find that there naturally exists a gap, which resembles the modality gap, between the prototype and image instance embeddings extracted from the frozen pre-trained backbone, and simply applying the same transformation during the adaptation phase constrains exploring the optimal representations and shrinks the gap between prototype and image representations. To solve this problem, we propose a simple yet effective method, contrastive prototype-image adaptation (CoPA), to adapt different transformations respectively for prototypes and images similarly to CLIP by treating prototypes as text prompts. Extensive experiments on Meta-Dataset demonstrate that CoPA achieves the state-of-the-art performance more efficiently. Meanwhile, further analyses also indicate that CoPA can learn better representation clusters, enlarge the gap, and achieve minimal validation loss at the enlarged gap.