Free Dynamics of Feature Learning Processes

TL;DR

This paper presents a statistical physics framework for understanding feature learning in regression models, deriving explicit error expressions and dynamical systems that describe the evolution of feature alignment during training.

Contribution

It introduces a novel dynamical systems approach to analyze the learning process of features, providing exact error formulas and insights into generalization conditions.

Findings

Explicit asymptotic expressions for train and test errors.

Identification of an effective ridge penalty as the train-test error ratio.

Dynamical equations describing the evolution of feature alignment.

Abstract

Regression models usually tend to recover a noisy signal in the form of a combination of regressors, also called features in machine learning, themselves being the result of a learning process.The alignment of the prior covariance feature matrix with the signal is known to play a key role in the generalization properties of the model, i.e. its ability to make predictions on unseen data during training. We present a statistical physics picture of the learning process. First we revisit the ridge regression to obtain compact asymptotic expressions for train and test errors, rendering manifest the conditions under which efficient generalization occurs. It is established thanks to an exact test-train sample error ratio combined with random matrix properties. Along the way in the form of a self-energy emerges an effective ridge penalty \textemdash\ precisely the train to test error ratio \textemdash\ which offer a very simple parameterization of the problem. This formulation appears convenient to tackle the learning process of the feature matrix itself. We derive an autonomous dynamical system in terms of elementary degrees of freedom of the problem determining the evolution of the relative alignment between the population matrix and the signal. A macroscopic counterpart of these equations is also obtained and various dynamical mechanisms are unveiled, allowing one to interpret the dynamics of simulated learning processes and reproduce trajectories of single experimental run with high precision.