Deep Learning of Compositional Targets with Hierarchical Spectral Methods

TL;DR

This paper investigates how depth in neural networks provides a computational advantage for learning compositional functions, demonstrating that multi-layer spectral methods can efficiently learn such targets in high-dimensional Gaussian settings.

Contribution

It introduces a spectral estimation approach for three-layer models that exploits compositional structure, showing a clear advantage over shallow methods in sample complexity.

Findings

Three-layer spectral methods outperform shallow estimators in learning compositional targets.

Sample complexity is sharply reduced with increased network depth.

Gaussian universality underpins the theoretical analysis.

Abstract

Why depth yields a genuine computational advantage over shallow methods remains a central open question in learning theory. We study this question in a controlled high-dimensional Gaussian setting, focusing on compositional target functions. We analyze their learnability using an explicit three-layer fitting model trained via layer-wise spectral estimators. Although the target is globally a high-degree polynomial, its compositional structure allows learning to proceed in stages: an intermediate representation reveals structure that is inaccessible at the input level. This reduces learning to simpler spectral estimation problems, well studied in the context of multi-index models, whereas any shallow estimator must resolve all components simultaneously. Our analysis relies on Gaussian universality, leading to sharp separations in sample complexity between two and three-layer learning strategies.