Deep Learning as Neural Low-Degree Filtering: A Spectral Theory of Hierarchical Feature Learning

TL;DR

This paper introduces Neural LoFi, a spectral theory framework for understanding hierarchical feature learning in deep neural networks, explaining how representations evolve layer by layer.

Contribution

It provides a mathematically explicit model for multi-layer feature learning beyond the lazy regime, connecting spectral procedures with deep learning dynamics.

Findings

Neural LoFi predicts layer-wise feature selection and concept emergence.

The framework explains how depth constructs new features from existing ones.

Experiments show Neural LoFi outperforms lazy random-feature baselines and aligns with real gradient-descent features.

Abstract

Understanding how deep neural networks learn useful internal representations from data remains a central open problem in the theory of deep learning. We introduce Neural Low-Degree Filtering (Neural LoFi), a stylized limit of gradient-based training in which hierarchical feature learning becomes an explicit iterative spectral procedure. In this limit, the dynamics at each layer decouple: given the current representation, the next layer selects directions with maximal accessible low-degree correlation to the label. This yields a tractable surrogate mechanism for deep learning, together with a natural kernel-space interpretation. Neural LoFi provides a mathematically explicit framework for studying multi-layer feature learning beyond the lazy regime. It predicts how representations are selected layer by layer, explains how emergence of concepts arises with given sample complexity,and gives a concrete mechanism by which depth progressively constructs new features from old ones through low-degree compositionality. We complement the theory with mechanistic experiments on fully connected and convolutional architectures, showing that Neural LoFi improves over lazy random-feature baselines, recovers meaningful structured filters, and predicts representations aligned with early gradient-descent feature discovery with real datasets.