A Noise Sensitivity Exponent Controls Large Statistical-to-Computational Gaps in Single- and Multi-Index Models

TL;DR

This paper introduces the Noise Sensitivity Exponent (NSE) as a key factor that determines the presence and size of statistical-to-computational gaps in high-dimensional single- and multi-index models, linking noise robustness and learning difficulty.

Contribution

The study reveals that the NSE, determined by the activation function, unifies understanding of computational hardness and feature learning in high-dimensional models.

Findings

NSE characterizes the onset of computational bottlenecks in single-index models with high noise.

NSE controls the transition to learnability in large separable multi-index models.

NSE determines the optimal rate of sequential feature learning in hierarchical multi-index models.

Abstract

Understanding when learning is statistically possible yet computationally hard is a central challenge in high-dimensional statistics. In this work, we investigate this question in the context of single- and multi-index models, classes of functions widely studied as benchmarks to probe the ability of machine learning methods to discover features in high-dimensional data. Our main contribution is to show that a Noise Sensitivity Exponent (NSE) - a simple quantity determined by the activation function - governs the existence and magnitude of statistical-to-computational gaps within a broad regime of these models. We first establish that, in single-index models with large additive noise, the onset of a computational bottleneck is fully characterized by the NSE. We then demonstrate that the same exponent controls a statistical-computational gap in the specialization transition of large separable multi-index models, where individual components become learnable. Finally, in hierarchical multi-index models, we show that the NSE governs the optimal computational rate in which different directions are sequentially learned. Taken together, our results identify the NSE as a unifying property linking noise robustness, computational hardness, and feature specialization in high-dimensional learning.