Active Learning for Single Neuron Models with Lipschitz Non-Linearities

TL;DR

This paper demonstrates that active learning strategies effective for linear models can be extended to single neuron models with Lipschitz non-linearities, providing strong approximation guarantees and empirical improvements over uniform sampling.

Contribution

It introduces a novel application of leverage score sampling for active learning of single neuron models with Lipschitz non-linearities, with theoretical guarantees and empirical validation.

Findings

Leverage score sampling outperforms uniform sampling in fitting single neuron models.

Strong approximation guarantees are achievable for Lipschitz non-linearities.

The approach is effective in the agnostic setting with adversarial noise.

Abstract

We consider the problem of active learning for single neuron models, also sometimes called ``ridge functions'', in the agnostic setting (under adversarial label noise). Such models have been shown to be broadly effective in modeling physical phenomena, and for constructing surrogate data-driven models for partial differential equations. Surprisingly, we show that for a single neuron model with any Lipschitz non-linearity (such as the ReLU, sigmoid, absolute value, low-degree polynomial, among others), strong provable approximation guarantees can be obtained using a well-known active learning strategy for fitting \emph{linear functions} in the agnostic setting. % -- i.e. for the case when there is no non-linearity. Namely, we can collect samples via statistical \emph{leverage score sampling}, which has been shown to be near-optimal in other active learning scenarios. We support our theoretical results with empirical simulations showing that our proposed active learning strategy based on leverage score sampling outperforms (ordinary) uniform sampling when fitting single neuron models.