How does feature learning reshape the function space?

TL;DR

This paper provides a detailed analysis of how feature learning in neural networks during training induces a data-adaptive kernel that reshapes the function space, emphasizing target-aligned directions.

Contribution

It offers a precise characterization of the evolution of the function space during gradient descent, revealing the induced kernel's role in reshaping spectral properties.

Findings

Post-gradient step feature distribution approximates a target-dependent spiked Gaussian covariance.

Feature learning introduces a data-adaptive kernel that amplifies target-aligned eigenvalues.

Gradient descent induces a deformation in the function space that enhances signal-aligned directions.

Abstract

Feature learning is widely regarded as the key mechanism distinguishing neural networks from fixed-kernel methods, yet its impact on the induced function space remains poorly understood. In this work, we precisely characterize how the function space spanned by the features of a two-layer neural network evolves during gradient descent training. We prove that, in the high-dimensional proportional regime, after a large gradient step the post-update feature distribution is well approximated by a target-dependent spiked Gaussian covariance. This induces a data-adaptive kernel that reshapes the function space and modifies its spectral structure. Our analysis reveals that feature learning can be interpreted as a distributional transformation in either parameter space or input space, equivalently as the introduction of a target-dependent kernel. In particular, it selectively amplifies eigenvalues aligned with the target direction and mixes leading eigenfunctions, coupling the top radial mode with a target-aligned quadratic harmonic. Overall, our results provide a precise function-space perspective on early-stage feature learning: rather than just rescaling a fixed kernel, gradient descent induces a data-adaptive deformation that preferentially enhances directions aligned with the signal in the data.