Few-shot Classification as Multi-instance Verification: Effective Backbone-agnostic Transfer across Domains

TL;DR

This paper introduces the MIV-head, a backbone-agnostic, efficient method for cross-domain few-shot classification that operates without fine-tuning, achieving competitive accuracy with lower adaptation costs.

Contribution

The paper proposes the MIV-head, a novel approach that handles few-shot domain adaptation without backbone fine-tuning, outperforming traditional classification heads and matching state-of-the-art adapter methods.

Findings

MIV-head achieves high accuracy across various backbones and domains.

It significantly reduces adaptation cost compared to fine-tuning methods.

The approach is effective with both CNN and vision transformer backbones.

Abstract

We investigate cross-domain few-shot learning under the constraint that fine-tuning of backbones (i.e., feature extractors) is impossible or infeasible -- a scenario that is increasingly common in practical use cases. Handling the low-quality and static embeddings produced by frozen, "black-box" backbones leads to a problem representation of few-shot classification as a series of multiple instance verification (MIV) tasks. Inspired by this representation, we introduce a novel approach to few-shot domain adaptation, named the "MIV-head", akin to a classification head that is agnostic to any pretrained backbone and computationally efficient. The core components designed for the MIV-head, when trained on few-shot data from a target domain, collectively yield strong performance on test data from that domain. Importantly, it does so without fine-tuning the backbone, and within the "meta-testing" phase. Experimenting under various settings and on an extension of the Meta-dataset benchmark for cross-domain few-shot image classification, using representative off-the-shelf convolutional neural network and vision transformer backbones pretrained on ImageNet1K, we show that the MIV-head achieves highly competitive accuracy when compared to state-of-the-art "adapter" (or partially fine-tuning) methods applied to the same backbones, while incurring substantially lower adaptation cost. We also find well-known "classification head" approaches lag far behind in terms of accuracy. Ablation study empirically justifies the core components of our approach. We share our code at https://github.com/xxweka/MIV-head.