Measurability in the Fundamental Theorem of Statistical Learning

TL;DR

This paper rigorously analyzes the measurability assumptions in the Fundamental Theorem of Statistical Learning, providing a measure-theoretic foundation and extending results to hypothesis spaces in o-minimal structures, including neural networks.

Contribution

It offers a measure-theoretic clarification of the theorem, establishing minimal assumptions and applying to neural networks with common activation functions.

Findings

Explicit measurability conditions for the theorem

Self-contained proof of the agnostic case

PAC learnability results for neural networks with ReLU and sigmoid

Abstract

The Fundamental Theorem of Statistical Learning states that a hypothesis space is PAC learnable if and only if its VC dimension is finite. For the agnostic model of PAC learning, the literature so far presents proofs of this theorem that often tacitly impose several measurability assumptions on the involved sets and functions. We scrutinize these proofs from a measure-theoretic perspective in order to explicitly extract the assumptions needed for a rigorous argument. This leads to a sound statement as well as a detailed and self-contained proof of the Fundamental Theorem of Statistical Learning in the agnostic setting, showcasing the minimal measurability requirements needed. As the Fundamental Theorem of Statistical Learning underpins a wide range of further theoretical developments, our results are of foundational importance: A careful analysis of measurability aspects is essential, especially when the theorem is used in settings where measure-theoretic subtleties play a role. We particularly discuss applications in Model Theory, considering NIP and o-minimal structures. Our main theorem presents sufficient conditions for the PAC learnability of hypothesis spaces defined over o-minimal expansions of the reals. This class of hypothesis spaces covers all artificial neural networks for binary classification that use commonly employed activation functions like ReLU and the sigmoid function.