On the Kolmogorov Complexity of Binary Classifiers

TL;DR

This paper establishes tight bounds on the expected Kolmogorov complexity of binary classifiers consistent with data, linking it to the complexity of the target concept and label entropy.

Contribution

It introduces precise bounds on classifier complexity, connecting Kolmogorov complexity with data and label properties, advancing theoretical understanding.

Findings

Expected classifier complexity is bounded by target concept complexity plus label entropy.

Provides tight upper and lower bounds on Kolmogorov complexity for classifiers.

Links classifier complexity to information-theoretic measures.

Abstract

We provide tight upper and lower bounds on the expected minimum Kolmogorov complexity of binary classifiers that are consistent with labeled samples. The expected size is not more than complexity of the target concept plus the conditional entropy of the labels given the sample.