Full-Batch Gradient Descent Outperforms One-Pass SGD: Sample Complexity Separation in Single-Index Learning

TL;DR

This paper demonstrates that full-batch gradient descent can outperform one-pass stochastic gradient descent in single-index models by requiring fewer samples for effective learning, especially when using truncated activations.

Contribution

It reveals that full-batch GD can achieve better sample complexity than online SGD in certain single-index models with quadratic activation, under specific conditions.

Findings

Full-batch GD outperforms one-pass SGD in sample efficiency for certain models.

Truncating the activation improves the optimization landscape for GD.

A trajectory analysis shows GD achieves strong recovery with fewer samples and steps.

Abstract

It is folklore that reusing training data more than once can improve the statistical efficiency of gradient-based learning. However, beyond linear regression, the theoretical advantage of full-batch gradient descent (GD, which always reuses all the data) over one-pass stochastic gradient descent (online SGD, which uses each data point only once) remains unclear. In this work, we consider learning a $d$-dimensional single-index model with a quadratic activation, for which it is known that one-pass SGD requires $n\gtrsim d\log d$ samples to achieve weak recovery. We first show that this $\log d$ factor in the sample complexity persists for full-batch spherical GD on the correlation loss; however, by simply truncating the activation, full-batch GD exhibits a favorable optimization landscape at $n \simeq d$ samples, thereby outperforming one-pass SGD (with the same activation) in statistical efficiency. We complement this result with a trajectory analysis of full-batch GD on the squared loss from small initialization, showing that $n \gtrsim d$ samples and $T \gtrsim\log d$ gradient steps suffice to achieve strong (exact) recovery.