Robust Meta-learning with Sampling Noise and Label Noise via Eigen-Reptile

TL;DR

This paper introduces Eigen-Reptile, a meta-learning method that effectively handles sampling and label noise by focusing on the main direction of historical task-specific parameters, improving robustness in few-shot learning scenarios.

Contribution

The paper proposes Eigen-Reptile, a novel meta-learning algorithm that updates meta-parameters using the main direction of historical parameters, and introduces ISPL to enhance accuracy with noisy labels.

Findings

Outperforms existing gradient-based meta-learning methods on various tasks.

Effectively mitigates overfitting caused by sampling noise.

Shows robustness against label noise in datasets.

Abstract

Recent years have seen a surge of interest in meta-learning techniques for tackling the few-shot learning (FSL) problem. However, the meta-learner is prone to overfitting since there are only a few available samples, which can be identified as sampling noise on a clean dataset. Moreover, when handling the data with noisy labels, the meta-learner could be extremely sensitive to label noise on a corrupted dataset. To address these two challenges, we present Eigen-Reptile (ER) that updates the meta-parameters with the main direction of historical task-specific parameters to alleviate sampling and label noise. Specifically, the main direction is computed in a fast way, where the scale of the calculated matrix is related to the number of gradient steps instead of the number of parameters. Furthermore, to obtain a more accurate main direction for Eigen-Reptile in the presence of many noisy labels, we further propose Introspective Self-paced Learning (ISPL). We have theoretically and experimentally demonstrated the soundness and effectiveness of the proposed Eigen-Reptile and ISPL. Particularly, our experiments on different tasks show that the proposed method is able to outperform or achieve highly competitive performance compared with other gradient-based methods with or without noisy labels. The code and data for the proposed method are provided for research purposes https://github.com/Anfeather/Eigen-Reptile.