Bridging Few-Shot Learning and Adaptation: New Challenges of Support-Query Shift

TL;DR

This paper introduces the problem of Few-Shot Learning under Support/Query Shift (FSQS), providing a testbed, analyzing the impact of distribution shifts on FSL algorithms, and proposing a new method combining Optimal Transport with Prototypical Networks to address this challenge.

Contribution

The paper presents a new FSQS testbed, evaluates existing FSL algorithms under distribution shifts, and introduces a novel OT-based method to improve adaptation in shifted scenarios.

Findings

FSL algorithms experience significant accuracy drops under FSQS.

Transductive algorithms can mitigate the effects of distribution shift.

Combining Optimal Transport with Prototypical Networks improves performance in FSQS.

Abstract

Few-Shot Learning (FSL) algorithms have made substantial progress in learning novel concepts with just a handful of labelled data. To classify query instances from novel classes encountered at test-time, they only require a support set composed of a few labelled samples. FSL benchmarks commonly assume that those queries come from the same distribution as instances in the support set. However, in a realistic set-ting, data distribution is plausibly subject to change, a situation referred to as Distribution Shift (DS). The present work addresses the new and challenging problem of Few-Shot Learning under Support/Query Shift (FSQS) i.e., when support and query instances are sampled from related but different distributions. Our contributions are the following. First, we release a testbed for FSQS, including datasets, relevant baselines and a protocol for a rigorous and reproducible evaluation. Second, we observe that well-established FSL algorithms unsurprisingly suffer from a considerable drop in accuracy when facing FSQS, stressing the significance of our study. Finally, we show that transductive algorithms can limit the inopportune effect of DS. In particular, we study both the role of Batch-Normalization and Optimal Transport (OT) in aligning distributions, bridging Unsupervised Domain Adaptation with FSL. This results in a new method that efficiently combines OT with the celebrated Prototypical Networks. We bring compelling experiments demonstrating the advantage of our method. Our work opens an exciting line of research by providing a testbed and strong baselines. Our code is available at https://github.com/ebennequin/meta-domain-shift.