BatMan-CLR: Making Few-shots Meta-Learners Resilient Against Label Noise

TL;DR

This paper analyzes how label noise affects meta-learning models and introduces two sampling techniques, BatMan and Man, to improve their robustness against noisy labels, significantly reducing accuracy drops.

Contribution

It provides the first extensive analysis of label noise impact on meta-learners and proposes novel sampling methods to enhance their resilience.

Findings

Meta-learners' accuracy drops up to 42% with label noise.

BatMan and Man techniques limit accuracy loss to around 2.5-9.4% under 60% label noise.

Proposed methods improve robustness across multiple datasets.

Abstract

The negative impact of label noise is well studied in classical supervised learning yet remains an open research question in meta-learning. Meta-learners aim to adapt to unseen learning tasks by learning a good initial model in meta-training and consecutively fine-tuning it according to new tasks during meta-testing. In this paper, we present the first extensive analysis of the impact of varying levels of label noise on the performance of state-of-the-art meta-learners, specifically gradient-based $N$-way $K$-shot learners. We show that the accuracy of Reptile, iMAML, and foMAML drops by up to 42% on the Omniglot and CifarFS datasets when meta-training is affected by label noise. To strengthen the resilience against label noise, we propose two sampling techniques, namely manifold (Man) and batch manifold (BatMan), which transform the noisy supervised learners into semi-supervised ones to increase the utility of noisy labels. We first construct manifold samples of $N$-way $2$-contrastive-shot tasks through augmentation, learning the embedding via a contrastive loss in meta-training, and then perform classification through zeroing on the embedding in meta-testing. We show that our approach can effectively mitigate the impact of meta-training label noise. Even with 60% wrong labels \batman and \man can limit the meta-testing accuracy drop to ${2.5}$, ${9.4}$, ${1.1}$ percent points, respectively, with existing meta-learners across the Omniglot, CifarFS, and MiniImagenet datasets.