On Single Index Models beyond Gaussian Data

TL;DR

This paper extends the understanding of single-index models beyond Gaussian data, showing that stochastic gradient descent can effectively recover the underlying direction even when data stability or symmetry assumptions are violated.

Contribution

It generalizes previous Gaussian-based results to broader data distributions, demonstrating efficient recovery of the model's key parameter under less restrictive conditions.

Findings

SGD recovers the true direction in high dimensions beyond Gaussian data

Recovery guarantees hold under relaxed stability and symmetry assumptions

Extends theoretical understanding of single-index models in non-Gaussian settings

Abstract

Sparse high-dimensional functions have arisen as a rich framework to study the behavior of gradient-descent methods using shallow neural networks, showcasing their ability to perform feature learning beyond linear models. Amongst those functions, the simplest are single-index models $f(x) = \phi( x \cdot \theta^*)$, where the labels are generated by an arbitrary non-linear scalar link function $\phi$ applied to an unknown one-dimensional projection $\theta^*$ of the input data. By focusing on Gaussian data, several recent works have built a remarkable picture, where the so-called information exponent (related to the regularity of the link function) controls the required sample complexity. In essence, these tools exploit the stability and spherical symmetry of Gaussian distributions. In this work, building from the framework of \cite{arous2020online}, we explore extensions of this picture beyond the Gaussian setting, where both stability or symmetry might be violated. Focusing on the planted setting where $\phi$ is known, our main results establish that Stochastic Gradient Descent can efficiently recover the unknown direction $\theta^*$ in the high-dimensional regime, under assumptions that extend previous works \cite{yehudai2020learning,wu2022learning}.