A universally consistent learning rule with a universally monotone error

TL;DR

This paper introduces a deterministic, universally consistent learning rule that guarantees a non-increasing expected error with increasing sample size across all data distributions, addressing a long-standing open problem.

Contribution

It proposes a new data-dependent partitioning learning rule that ensures monotone error reduction, solving the existence question posed in 1996.

Findings

Expected error is monotone non-increasing with sample size.

The rule is fully deterministic and applicable in arbitrary domains.

It avoids regions of convex error by selective partitioning.

Abstract

We present a universally consistent learning rule whose expected error is monotone non-increasing with the sample size under every data distribution. The question of existence of such rules was brought up in 1996 by Devroye, Gy\"orfi and Lugosi (who called them "smart"). Our rule is fully deterministic, a data-dependent partitioning rule constructed in an arbitrary domain (a standard Borel space) using a cyclic order. The central idea is to only partition at each step those cyclic intervals that exhibit a sufficient empirical diversity of labels, thus avoiding a region where the error function is convex.