The Generative Leap: Sharp Sample Complexity for Efficiently Learning Gaussian Multi-Index Models

TL;DR

This paper introduces the generative leap exponent for Gaussian multi-index models, establishing optimal sample complexity bounds and proposing an agnostic spectral U-statistic method for efficient hidden subspace estimation.

Contribution

It extends the concept of the generative exponent to multi-index models, providing matching lower and upper bounds on sample complexity and a novel estimation procedure.

Findings

Sample complexity of $\Theta(d^{1 \vee \k/2})$ is necessary and sufficient.

The proposed spectral U-statistic estimator is agnostic and effective.

The generative leap exponent is computed for deep neural network examples.

Abstract

In this work we consider generic Gaussian Multi-index models, in which the labels only depend on the (Gaussian) $d$-dimensional inputs through their projection onto a low-dimensional $r = O_d(1)$ subspace, and we study efficient agnostic estimation procedures for this hidden subspace. We introduce the \emph{generative leap} exponent $k^\star$, a natural extension of the generative exponent from [Damian et al.'24] to the multi-index setting. We first show that a sample complexity of $n=\Theta(d^{1 \vee \k/2})$ is necessary in the class of algorithms captured by the Low-Degree-Polynomial framework. We then establish that this sample complexity is also sufficient, by giving an agnostic sequential estimation procedure (that is, requiring no prior knowledge of the multi-index model) based on a spectral U-statistic over appropriate Hermite tensors. We further compute the generative leap exponent for several examples including piecewise linear functions (deep ReLU networks with bias), and general deep neural networks (with $r$-dimensional first hidden layer).