The One-Inclusion Graph Algorithm is not Always Optimal

TL;DR

This paper demonstrates that the one-inclusion graph algorithm, while optimal in expectation, does not always guarantee optimal high-probability risk bounds in PAC classification, challenging prior assumptions and highlighting limitations of certain prediction strategies.

Contribution

It refutes Warmuth's conjecture by providing counterexamples using error-correcting codes, showing the gap between expected and high-probability performance in PAC algorithms.

Findings

The one-inclusion graph algorithm's high-probability bounds are limited by Markov's inequality.

Counterexamples are constructed using Varshamov-Tenengolts codes.

The discrepancy between expected optimality and high-probability performance applies to recent algorithms.

Abstract

The one-inclusion graph algorithm of Haussler, Littlestone, and Warmuth achieves an optimal in-expectation risk bound in the standard PAC classification setup. In one of the first COLT open problems, Warmuth conjectured that this prediction strategy always implies an optimal high probability bound on the risk, and hence is also an optimal PAC algorithm. We refute this conjecture in the strongest sense: for any practically interesting Vapnik-Chervonenkis class, we provide an in-expectation optimal one-inclusion graph algorithm whose high probability risk bound cannot go beyond that implied by Markov's inequality. Our construction of these poorly performing one-inclusion graph algorithms uses Varshamov-Tenengolts error correcting codes. Our negative result has several implications. First, it shows that the same poor high-probability performance is inherited by several recent prediction strategies based on generalizations of the one-inclusion graph algorithm. Second, our analysis shows yet another statistical problem that enjoys an estimator that is provably optimal in expectation via a leave-one-out argument, but fails in the high-probability regime. This discrepancy occurs despite the boundedness of the binary loss for which arguments based on concentration inequalities often provide sharp high probability risk bounds.