Gradient-Based Feature Learning under Structured Data

TL;DR

This paper explores how structured data, specifically spiked covariance models, influences the effectiveness and sample complexity of gradient-based feature learning, revealing new phenomena and improvements over isotropic assumptions.

Contribution

It demonstrates the impact of data structure on gradient dynamics, introduces normalization techniques to improve recovery, and shows reduced sample complexity in structured settings.

Findings

Gradient dynamics may fail in anisotropic data without normalization.

Normalization techniques can improve direction recovery.

Structured data can reduce sample complexity and outperform kernel methods.

Abstract

Recent works have demonstrated that the sample complexity of gradient-based learning of single index models, i.e. functions that depend on a 1-dimensional projection of the input data, is governed by their information exponent. However, these results are only concerned with isotropic data, while in practice the input often contains additional structure which can implicitly guide the algorithm. In this work, we investigate the effect of a spiked covariance structure and reveal several interesting phenomena. First, we show that in the anisotropic setting, the commonly used spherical gradient dynamics may fail to recover the true direction, even when the spike is perfectly aligned with the target direction. Next, we show that appropriate weight normalization that is reminiscent of batch normalization can alleviate this issue. Further, by exploiting the alignment between the (spiked) input covariance and the target, we obtain improved sample complexity compared to the isotropic case. In particular, under the spiked model with a suitably large spike, the sample complexity of gradient-based training can be made independent of the information exponent while also outperforming lower bounds for rotationally invariant kernel methods.