Statistical-Computational Trade-offs in Learning Multi-Index Models via Harmonic Analysis

TL;DR

This paper analyzes the fundamental limits and algorithms for learning multi-index models using harmonic analysis, providing sharp complexity bounds and spectral methods that adapt to different trade-offs between data and computation.

Contribution

It offers a harmonic-analytic framework for characterizing learning complexity in multi-index models, extending previous Gaussian-specific results and proposing spectral algorithms based on harmonic tensor unfolding.

Findings

Derived sharp statistical and computational lower bounds for MIMs

Constructed spectral algorithms that nearly achieve these bounds

Demonstrated flexible trade-offs between sample size and runtime

Abstract

We study the problem of learning multi-index models (MIMs), where the label depends on the input $\boldsymbol{x} \in \mathbb{R}^d$ only through an unknown $\mathsf{s}$-dimensional projection $\boldsymbol{W}_*^\mathsf{T} \boldsymbol{x} \in \mathbb{R}^\mathsf{s}$. Exploiting the equivariance of this problem under the orthogonal group $\mathcal{O}_d$, we obtain a sharp harmonic-analytic characterization of the learning complexity for MIMs with spherically symmetric inputs -- which refines and generalizes previous Gaussian-specific analyses. Specifically, we derive statistical and computational complexity lower bounds within the Statistical Query (SQ) and Low-Degree Polynomial (LDP) frameworks. These bounds decompose naturally across spherical harmonic subspaces. Guided by this decomposition, we construct a family of spectral algorithms based on harmonic tensor unfolding that sequentially recover the latent directions and (nearly) achieve these SQ and LDP lower bounds. Depending on the choice of harmonic degree sequence, these estimators can realize a broad range of trade-offs between sample and runtime complexity. From a technical standpoint, our results build on the semisimple decomposition of the $\mathcal{O}_d$-action on $L^2 (\mathbb{S}^{d-1})$ and the intertwining isomorphism between spherical harmonics and traceless symmetric tensors.