Robustly Learning Monotone Single-Index Models

TL;DR

This paper introduces the first efficient algorithm for learning monotone single-index models with adversarial noise, achieving a constant factor approximation for a broad class of monotone functions under Gaussian distribution.

Contribution

It presents a novel optimization framework that moves beyond traditional gradient methods to effectively learn all monotone activations with bounded moments, including discontinuous functions.

Findings

First computationally efficient algorithm for this task.

Achieves constant factor approximation for all monotone activations.

Works under adversarial label noise and Gaussian distribution.

Abstract

We consider the basic problem of learning Single-Index Models with respect to the square loss under the Gaussian distribution in the presence of adversarial label noise. Our main contribution is the first computationally efficient algorithm for this learning task, achieving a constant factor approximation, that succeeds for the class of {\em all} monotone activations with bounded moment of order $2 + \zeta,$ for $\zeta > 0.$ This class in particular includes all monotone Lipschitz functions and even discontinuous functions like (possibly biased) halfspaces. Prior work for the case of unknown activation either does not attain constant factor approximation or succeeds for a substantially smaller family of activations. The main conceptual novelty of our approach lies in developing an optimization framework that steps outside the boundaries of usual gradient methods and instead identifies a useful vector field to guide the algorithm updates by directly leveraging the problem structure, properties of Gaussian spaces, and regularity of monotone functions.