Leave-One-Out Learning with Log-Loss

TL;DR

This paper investigates leave-one-out regret in batch learning with log-loss for deterministic outcomes, establishing minimax regret bounds and demonstrating the learnability of various hypothesis classes in the individual setting.

Contribution

It introduces a leave-one-out regret criterion, derives minimax bounds for multinomial and VC classes, and proves universal learnability with log-loss in the individual setting.

Findings

Minimax regret for multinomial over m symbols is (m-1)/N.

VC dimension d classes are learnable with regret O(d log N / N).

Universal batch learning with log-loss is achievable in the individual setting.

Abstract

We study batch learning with log-loss in the individual setting, where the outcome sequence is deterministic. Because empirical statistics are not directly applicable in this regime, obtaining regret guarantees for batch learning has long posed a fundamental challenge. We propose a natural criterion based on leave-one-out regret and analyze its minimax value for several hypothesis classes. For the multinomial simplex over $m$ symbols, we show that the minimax regret is $\frac{m-1}{N} + o\!\left(\frac{1}{N}\right)$, and compare it to the stochastic realizable case where it is $\frac{m-1}{2N} + o\!\left(\frac{1}{N}\right)$. More generally, we prove that every hypothesis class of VC dimension $d$ is learnable in the individual batch-learning problem, with regret at most $\frac{d\log(N)}{N} + o\!\left(\frac{\log(N)}{N}\right)$, and we establish matching lower bounds for certain classes. We further derive additional upper bounds that depend on structural properties of the hypothesis class. These results establish, for the first time, that universal batch learning with log-loss is possible in the individual setting.