Limitations of SGD for Multi-Index Models Beyond Statistical Queries

TL;DR

This paper investigates the limitations of standard SGD in learning multi-index models, highlighting the inadequacy of the SQ framework and proposing a new approach applicable to various models including neural networks.

Contribution

It introduces a novel non-SQ framework to analyze vanilla SGD's limitations in multi-index models, extending understanding beyond prior SQ-based analyses.

Findings

SQ framework can be misleading for SGD analysis

Standard SGD faces fundamental limitations in multi-index models

Results apply to deep neural network architectures

Abstract

Understanding the limitations of gradient methods, and stochastic gradient descent (SGD) in particular, is a central challenge in learning theory. To that end, a commonly used tool is the Statistical Queries (SQ) framework, which studies performance limits of algorithms based on noisy interaction with the data. However, it is known that the formal connection between the SQ framework and SGD is tenuous: Existing results typically rely on adversarial or specially-structured gradient noise that does not reflect the noise in standard SGD, and (as we point out here) can sometimes lead to incorrect predictions. Moreover, many analyses of SGD for challenging problems rely on non-trivial algorithmic modifications, such as restricting the SGD trajectory to the sphere or using very small learning rates. To address these shortcomings, we develop a new, non-SQ framework to study the limitations of standard vanilla SGD, for single-index and multi-index models (namely, when the target function depends on a low-dimensional projection of the inputs). Our results apply to a broad class of settings and architectures, including (potentially deep) neural networks.