Low-dimensional Functions are Efficiently Learnable under Randomly Biased Distributions

TL;DR

This paper demonstrates that adding a small random shift to data distributions makes complex high-dimensional models, like single and multi index models, as easy to learn as simple linear functions, revealing their rarity.

Contribution

It proves that high complexity models become easy to learn under slight distribution perturbations, extending results to sparse Boolean functions and Juntas.

Findings

Random perturbations simplify learning complexity.

High complexity models are rare under perturbed distributions.

Single and multi index models become linearly learnable with small shifts.

Abstract

The problem of learning single index and multi index models has gained significant interest as a fundamental task in high-dimensional statistics. Many recent works have analysed gradient-based methods, particularly in the setting of isotropic data distributions, often in the context of neural network training. Such studies have uncovered precise characterisations of algorithmic sample complexity in terms of certain analytic properties of the target function, such as the leap, information, and generative exponents. These properties establish a quantitative separation between low and high complexity learning tasks. In this work, we show that high complexity cases are rare. Specifically, we prove that introducing a small random perturbation to the data distribution--via a random shift in the first moment--renders any Gaussian single index model as easy to learn as a linear function. We further extend this result to a class of multi index models, namely sparse Boolean functions, also known as Juntas.