Optimal Spectral Transitions in High-Dimensional Multi-Index Models

TL;DR

This paper introduces spectral algorithms for high-dimensional multi-index models, achieving optimal sample complexity for subspace reconstruction and revealing phase transitions similar to BBP in random matrix theory.

Contribution

It develops spectral methods that reach the theoretical limits of reconstructing index subspaces in multi-index models, extending beyond single-index cases.

Findings

Spectral algorithms achieve the optimal reconstruction threshold.

Above the threshold, the leading eigenvector aligns with the index subspace.

The phase transition resembles the BBP transition in random matrix theory.

Abstract

We consider the problem of how many samples from a Gaussian multi-index model are required to weakly reconstruct the relevant index subspace. Despite its increasing popularity as a testbed for investigating the computational complexity of neural networks, results beyond the single-index setting remain elusive. In this work, we introduce spectral algorithms based on the linearization of a message passing scheme tailored to this problem. Our main contribution is to show that the proposed methods achieve the optimal reconstruction threshold. Leveraging a high-dimensional characterization of the algorithms, we show that above the critical threshold the leading eigenvector correlates with the relevant index subspace, a phenomenon reminiscent of the Baik-Ben Arous-Peche (BBP) transition in spiked models arising in random matrix theory. Supported by numerical experiments and a rigorous theoretical framework, our work bridges critical gaps in the computational limits of weak learnability in multi-index model.