Near-Optimal Algorithms for Omniprediction

TL;DR

This paper introduces near-optimal algorithms for omniprediction, achieving low regret in online settings and efficient offline learning for a broad class of loss functions, advancing the theoretical understanding of omnipredictors.

Contribution

The work provides the first near-optimal online and offline algorithms for omniprediction with regret bounds matching single-loss minimization, and extends to a wide class of loss functions.

Findings

Online algorithm achieves $ ilde O ( oot{T} ext{log} |H|)$ regret.

Offline algorithm leverages ERM oracle for near-linear complexity.

Results apply to broad loss classes including Lipschitz and Bounded Variation losses.

Abstract

Omnipredictors are simple prediction functions that encode loss-minimizing predictions with respect to a hypothesis class $H$, simultaneously for every loss function within a class of losses $L$. In this work, we give near-optimal learning algorithms for omniprediction, in both the online and offline settings. To begin, we give an oracle-efficient online learning algorithm that acheives $(L,H)$-omniprediction with $\tilde O (\sqrt{T \log |H|})$ regret for any class of Lipschitz loss functions $L \subseteq L_\mathrm{Lip}$. Quite surprisingly, this regret bound matches the optimal regret for \emph{minimization of a single loss function} (up to a $\sqrt{\log(T)}$ factor). Given this online algorithm, we develop an online-to-offline conversion that achieves near-optimal complexity across a number of measures. In particular, for all bounded loss functions within the class of Bounded Variation losses $L_\mathrm{BV}$ (which include all convex, all Lipschitz, and all proper losses) and any (possibly-infinite) $H$, we obtain an offline learning algorithm that, leveraging an (offline) ERM oracle and $m$ samples from $D$, returns an efficient $(L_{\mathrm{BV}},H,\epsilon(m))$-omnipredictor for $\varepsilon(m)$ scaling near-linearly in the Rademacher complexity of a class derived from $H$ by taking convex combinations of a fixed number of elements of $\mathrm{Th} \circ H$.