On sensitivity of meta-learning to support data

TL;DR

This paper reveals that modern meta-learning algorithms for few-shot image classification are highly sensitive to support data, with accuracy varying drastically depending on the specific support images used, highlighting the need for larger class margins for robustness.

Contribution

The study uncovers the extreme sensitivity of meta-learning algorithms to support data and links this to class margins, proposing a direction for more robust meta-learning.

Findings

Accuracy can vary from 4% to 95% depending on support data.

Sensitivity is linked to class margins in the data.

Larger margins may improve robustness of meta-learning algorithms.

Abstract

Meta-learning algorithms are widely used for few-shot learning. For example, image recognition systems that readily adapt to unseen classes after seeing only a few labeled examples. Despite their success, we show that modern meta-learning algorithms are extremely sensitive to the data used for adaptation, i.e. support data. In particular, we demonstrate the existence of (unaltered, in-distribution, natural) images that, when used for adaptation, yield accuracy as low as 4\% or as high as 95\% on standard few-shot image classification benchmarks. We explain our empirical findings in terms of class margins, which in turn suggests that robust and safe meta-learning requires larger margins than supervised learning.