Multiclass Learnability Does Not Imply Sample Compression

TL;DR

This paper demonstrates that, unlike binary classes, multiclass hypothesis classes with finite DS dimension do not necessarily admit small sample compression schemes, highlighting a fundamental difference in learnability properties.

Contribution

It proves that finite DS dimension in multiclass classes does not guarantee the existence of finite sample compression schemes, contrasting with binary class results.

Findings

Binary classes admit size-bounded sample compression schemes

Multiclass classes with finite DS dimension do not necessarily admit such schemes

Highlights a fundamental difference between binary and multiclass learnability

Abstract

A hypothesis class admits a sample compression scheme, if for every sample labeled by a hypothesis from the class, it is possible to retain only a small subsample, using which the labels on the entire sample can be inferred. The size of the compression scheme is an upper bound on the size of the subsample produced. Every learnable binary hypothesis class (which must necessarily have finite VC dimension) admits a sample compression scheme of size only a finite function of its VC dimension, independent of the sample size. For multiclass hypothesis classes, the analog of VC dimension is the DS dimension. We show that the analogous statement pertaining to sample compression is not true for multiclass hypothesis classes: every learnable multiclass hypothesis class, which must necessarily have finite DS dimension, does not admit a sample compression scheme of size only a finite function of its DS dimension.