Joint Learning in the Gaussian Single Index Model

TL;DR

This paper investigates the joint learning of a projection and a univariate function in high-dimensional Gaussian models, providing theoretical convergence analysis and practical algorithms for efficient estimation.

Contribution

It introduces a novel analysis of gradient flow dynamics for joint learning in Gaussian single index models, including convergence guarantees and practical RKHS-based methods.

Findings

Gradient flow converges even with negatively correlated initial directions.

Convergence rate depends on the Gaussian regularity of the target function.

RKHS-based implementation enables efficient practical estimation.

Abstract

We consider the problem of jointly learning a one-dimensional projection and a univariate function in high-dimensional Gaussian models. Specifically, we study predictors of the form $f(x)=\varphi^\star(\langle w^\star, x \rangle)$, where both the direction $w^\star \in \mathcal{S}_{d-1}$, the sphere of $\mathbb{R}^d$, and the function $\varphi^\star: \mathbb{R} \to \mathbb{R}$ are learned from Gaussian data. This setting captures a fundamental non-convex problem at the intersection of representation learning and nonlinear regression. We analyze the gradient flow dynamics of a natural alternating scheme and prove convergence, with a rate controlled by the information exponent reflecting the \textit{Gaussian regularity} of the function $\varphi^\star$. Strikingly, our analysis shows that convergence still occurs even when the initial direction is negatively correlated with the target. On the practical side, we demonstrate that such joint learning can be effectively implemented using a Reproducing Kernel Hilbert Space (RKHS) adapted to the structure of the problem, enabling efficient and flexible estimation of the univariate function. Our results offer both theoretical insight and practical methodology for learning low-dimensional structure in high-dimensional settings.