Learning single-index models via harmonic decomposition

TL;DR

This paper introduces a novel approach to learning single-index models using spherical harmonics, providing a new perspective that captures rotational symmetry and characterizes complexity under various input distributions.

Contribution

It proposes the use of spherical harmonics instead of Hermite polynomials for analyzing single-index models, and develops estimators with optimal sample complexity or runtime.

Findings

Characterizes learning complexity under arbitrary spherically symmetric distributions.

Introduces tensor unfolding and online SGD estimators with optimal properties.

Recovers and extends existing results for Gaussian inputs.

Abstract

We study the problem of learning single-index models, where the label $y \in \mathbb{R}$ depends on the input $\boldsymbol{x} \in \mathbb{R}^d$ only through an unknown one-dimensional projection $\langle \boldsymbol{w}_*,\boldsymbol{x}\rangle$. Prior work has shown that under Gaussian inputs, the statistical and computational complexity of recovering $\boldsymbol{w}_*$ is governed by the Hermite expansion of the link function. In this paper, we propose a new perspective: we argue that $spherical$ $harmonics$ -- rather than $Hermite$ $polynomials$ -- provide the natural basis for this problem, as they capture its intrinsic $rotational$ $symmetry$. Building on this insight, we characterize the complexity of learning single-index models under arbitrary spherically symmetric input distributions. We introduce two families of estimators -- based on tensor unfolding and online SGD -- that respectively achieve either optimal sample complexity or optimal runtime, and argue that estimators achieving both may not exist in general. When specialized to Gaussian inputs, our theory not only recovers and clarifies existing results but also reveals new phenomena that had previously been overlooked.