Sample compression schemes for VC classes

TL;DR

This paper proves that any concept class with VC dimension d has a sample compression scheme of size exponential in d, linking compression schemes to VC theory and PAC learnability.

Contribution

It establishes that all VC classes admit exponential-size sample compression schemes, answering a longstanding question in learning theory.

Findings

Every VC class has a sample compression scheme of size exponential in VC dimension.

The proof involves an approximate minimax phenomenon for low VC dimension binary matrices.

Results connect sample compression, VC theory, and game theory concepts.

Abstract

Sample compression schemes were defined by Littlestone and Warmuth (1986) as an abstraction of the structure underlying many learning algorithms. Roughly speaking, a sample compression scheme of size $k$ means that given an arbitrary list of labeled examples, one can retain only $k$ of them in a way that allows to recover the labels of all other examples in the list. They showed that compression implies PAC learnability for binary-labeled classes, and asked whether the other direction holds. We answer their question and show that every concept class $C$ with VC dimension $d$ has a sample compression scheme of size exponential in $d$. The proof uses an approximate minimax phenomenon for binary matrices of low VC dimension, which may be of interest in the context of game theory.