Meta-Learning Requires Meta-Augmentation

TL;DR

This paper identifies overfitting issues in meta-learning, introduces meta-augmentation as a technique to improve generalization, and demonstrates its effectiveness alongside existing regularization methods.

Contribution

It introduces meta-augmentation, an information-theoretic approach to reduce overfitting in meta-learning, enhancing generalization beyond existing regularization techniques.

Findings

Meta-overfitting occurs in common benchmarks.

Meta-augmentation reduces overfitting effectively.

Combining meta-augmentation with other regularizers yields significant improvements.

Abstract

Meta-learning algorithms aim to learn two components: a model that predicts targets for a task, and a base learner that quickly updates that model when given examples from a new task. This additional level of learning can be powerful, but it also creates another potential source for overfitting, since we can now overfit in either the model or the base learner. We describe both of these forms of metalearning overfitting, and demonstrate that they appear experimentally in common meta-learning benchmarks. We then use an information-theoretic framework to discuss meta-augmentation, a way to add randomness that discourages the base learner and model from learning trivial solutions that do not generalize to new tasks. We demonstrate that meta-augmentation produces large complementary benefits to recently proposed meta-regularization techniques.