Dynamics of Meta-learning Representation in the Teacher-student Scenario

TL;DR

This paper analyzes the learning dynamics of gradient-based meta-learning in neural networks, revealing how shared representations form and generalize across tasks using statistical physics methods.

Contribution

It provides a theoretical framework for understanding the formation of shared representations in meta-learning, focusing on nonlinear neural networks in a teacher-student setting.

Findings

Shared representations emerge during meta-training.

Hyperparameters significantly influence learning dynamics.

Theoretical insights into generalization in meta-learning.

Abstract

Gradient-based meta-learning algorithms have gained popularity for their ability to train models on new tasks using limited data. Empirical observations indicate that such algorithms are able to learn a shared representation across tasks, which is regarded as a key factor in their success. However, the in-depth theoretical understanding of the learning dynamics and the origin of the shared representation remains underdeveloped. In this work, we investigate the meta-learning dynamics of nonlinear two-layer neural networks trained on streaming tasks in the teacher-student scenario. Through the lens of statistical physics analysis, we characterize the macroscopic behavior of the meta-training processes, the formation of the shared representation, and the generalization ability of the model on new tasks. The analysis also points to the importance of the choice of certain hyperparameters of the learning algorithms.