Rethinking Meta-Learning from a Learning Lens

TL;DR

This paper reexamines meta-learning from a learning perspective, identifying issues like underfitting in current models, and introduces TRLearner, a method that uses task relations to improve meta-learning performance.

Contribution

It proposes a new perspective on meta-learning by modeling model components separately and introduces TRLearner, a relation-aware regularization method to enhance learning.

Findings

TRLearner effectively leverages task relations for better meta-learning.

Theoretical analysis shows mutual reinforcement among task-adapted models.

Empirical results demonstrate improved performance over baseline methods.

Abstract

Meta-learning seeks to learn a well-generalized model initialization from training tasks to solve unseen tasks. From the "learning to learn" perspective, the quality of the initialization is modeled with one-step gradient decent in the inner loop. However, contrary to theoretical expectations, our empirical analysis reveals that this may expose meta-learning to underfitting. To bridge the gap between theoretical understanding and practical implementation, we reconsider meta-learning from the "Learning" lens. We propose that the meta-learning model comprises two interrelated components: parameters for model initialization and a meta-layer for task-specific fine-tuning. These components will lead to the risks of overfitting and underfitting depending on tasks, and their solutions, fewer parameters vs. more meta-layer, are often in conflict. To address this, we aim to regulate the task information the model receives without modifying the data or model structure. Our theoretical analysis indicates that models adapted to different tasks can mutually reinforce each other, highlighting the effective information. Based on this insight, we propose TRLearner, a plug-and-play method that leverages task relation to calibrate meta-learning. It first extracts task relation matrices and then applies relation-aware consistency regularization to guide optimization. Extensive theoretical and empirical evaluations demonstrate its effectiveness.