A Theory of Universal Agnostic Learning

TL;DR

This paper develops a comprehensive theory for the optimal rates of binary classification in the agnostic setting, removing the realizability assumption and classifying concept classes into four fundamental rate categories.

Contribution

It extends previous realizable-case theories to the agnostic setting and introduces a tetrachotomy classification for the optimal convergence rates based on combinatorial structures.

Findings

Identifies four fundamental categories of convergence rates.

Provides a complete characterization of optimal rates for any concept class.

Links combinatorial structures to the classification of rate categories.

Abstract

We provide a complete theory of optimal universal rates for binary classification in the agnostic setting. This extends the realizable-case theory of Bousquet, Hanneke, Moran, van Handel, and Yehudayoff (2021) by removing the realizability assumption on the distribution. We identify a fundamental tetrachotomy of optimal rates: for every concept class, the optimal universal rate of convergence of the excess error rate is one of $e^{-n}$, $e^{-o(n)}$, $o(n^{-1/2})$, or arbitrarily slow. We further identify simple combinatorial structures which determine which of these categories any given concept class falls into.