Local EGOP for Continuous Index Learning

TL;DR

This paper introduces Local EGOP learning, a recursive algorithm for continuous index learning that adapts to local function variability, achieving efficient high-dimensional noise handling and improved regression performance.

Contribution

The paper presents a novel Local EGOP algorithm that adapts to local function structure using EGOP, enabling efficient high-dimensional noise learning and improved regression results.

Findings

Achieves intrinsic dimensional learning rates under noisy manifold conditions.

Outperforms deep learning in continuous single-index regression tasks.

Demonstrates effective local adaptation to function regularity.

Abstract

We introduce the setting of continuous index learning, in which a function of many variables varies only along a small number of directions at each point. For efficient estimation, it is beneficial for a learning algorithm to adapt, near each point $x$, to the subspace that captures the local variability of the function $f$. We pose this task as kernel adaptation along a manifold with noise, and introduce Local EGOP learning, a recursive algorithm that utilizes the Expected Gradient Outer Product (EGOP) quadratic form as both a metric and inverse-covariance of our target distribution. We prove that Local EGOP learning adapts to the regularity of the function of interest, showing that under a supervised noisy manifold hypothesis, intrinsic dimensional learning rates are achieved for arbitrarily high-dimensional noise. Empirically, we compare our algorithm to the feature learning capabilities of deep learning. Additionally, we demonstrate improved regression quality compared to two-layer neural networks in the continuous single-index setting.