Robust Empirical Risk Minimization with Tolerance

TL;DR

This paper investigates the limitations of empirical risk minimization in robust learning and proposes a tolerant variant that achieves sample-efficient robust classification under certain geometric conditions.

Contribution

It introduces a tolerant robust learning framework and demonstrates its effectiveness for VC classes with geometric niceness, requiring fewer samples than traditional methods.

Findings

Tolerant RERM can learn VC classes with geometric niceness.

Sample complexity is significantly reduced under the tolerant model.

Traditional RERM fails in robust settings, but the tolerant variant succeeds.

Abstract

Developing simple, sample-efficient learning algorithms for robust classification is a pressing issue in today's tech-dominated world, and current theoretical techniques requiring exponential sample complexity and complicated improper learning rules fall far from answering the need. In this work we study the fundamental paradigm of (robust) $\textit{empirical risk minimization}$ (RERM), a simple process in which the learner outputs any hypothesis minimizing its training error. RERM famously fails to robustly learn VC classes (Montasser et al., 2019a), a bound we show extends even to `nice' settings such as (bounded) halfspaces. As such, we study a recent relaxation of the robust model called $\textit{tolerant}$ robust learning (Ashtiani et al., 2022) where the output classifier is compared to the best achievable error over slightly larger perturbation sets. We show that under geometric niceness conditions, a natural tolerant variant of RERM is indeed sufficient for $\gamma$-tolerant robust learning VC classes over $\mathbb{R}^d$, and requires only $\tilde{O}\left( \frac{VC(H)d\log \frac{D}{\gamma\delta}}{\epsilon^2}\right)$ samples for robustness regions of (maximum) diameter $D$.