Realizable Bayes-Consistency for General Metric Losses

TL;DR

This paper characterizes the conditions under which a hypothesis class can achieve universal Bayes-consistency in the realizable setting for general metric losses, extending classical results beyond 0-1 classification.

Contribution

It provides a necessary and sufficient combinatorial condition, involving infinite non-decreasing Littlestone trees, for distribution-free learning rules to be universally consistent with metric losses.

Findings

Identifies the exact conditions for universal Bayes-consistency in metric loss settings.

Introduces the concept of infinite non-decreasing Littlestone trees for this characterization.

Extends classical Littlestone tree structures to more general loss functions.

Abstract

We study strong universal Bayes-consistency in the realizable setting for learning with general metric losses, extending classical characterizations beyond $0$-$1$ classification (Bousquet et al., 2020; Hanneke et al., 2021) and real-valued regression (Attias et al., 2024). Given an instance space $(X,\rho)$, a label space $(Y,\ell)$ with possibly unbounded loss, and a hypothesis class $H \subseteq Y^{X}$, we resolve the realizable case of an open problem presented in Tsir Cohen and Kontorovich (2022). Specifically, we find the necessary and sufficient conditions on the hypothesis class $H$ under which there exists a distribution-free learning rule whose risk converges almost surely to the best-in-class risk (which is zero) for every realizable data-generating distribution. Our main contribution is this sharp characterization in terms of a combinatorial obstruction: Similarly to Attias et al. (2024), we introduce the notion of an infinite non-decreasing $(\gamma_k)$-Littlestone tree, where $\gamma_k \to \infty$. This extends the Littlestone tree structure used in Bousquet et al. (2020) to the metric loss setting.