Falsification and future performance

TL;DR

This paper reformulates capacity measures in statistical learning theory using information theory, linking the falsification of hypotheses to future predictor performance and connecting VC-entropy to message length in optimal coding.

Contribution

It introduces an information-theoretic perspective on VC-entropy and Rademacher complexity, relating them to hypothesis falsification and message length in learning.

Findings

Capacity measures count hypotheses falsified by the learning algorithm.

Future performance is partly determined by the number of falsified hypotheses.

Empirical VC-entropy relates to message length of the true hypothesis.

Abstract

We information-theoretically reformulate two measures of capacity from statistical learning theory: empirical VC-entropy and empirical Rademacher complexity. We show these capacity measures count the number of hypotheses about a dataset that a learning algorithm falsifies when it finds the classifier in its repertoire minimizing empirical risk. It then follows from that the future performance of predictors on unseen data is controlled in part by how many hypotheses the learner falsifies. As a corollary we show that empirical VC-entropy quantifies the message length of the true hypothesis in the optimal code of a particular probability distribution, the so-called actual repertoire.