Information-theoretic Limits of Learning and Estimation

TL;DR

This chapter explores the fundamental limits of learning and estimation using information theory, covering tools like concentration inequalities, metric entropy, and mutual information, with bounds on generalization and minimax risk.

Contribution

It provides a comprehensive introduction to information-theoretic bounds in learning and estimation, connecting various complexity measures and establishing fundamental limits.

Findings

Derived upper bounds on generalization error using metric entropy and mutual information.

Established lower bounds on minimax risk via Fano's inequality and covering numbers.

Linked information-theoretic tools to learning and estimation limits.

Abstract

Information theory plays a central role in establishing fundamental limits on what any learning or estimation algorithm can -- and cannot -- achieve, regardless of computational power. In this chapter, we provide an introduction to these connections. End-of-chapter exercises makes the material suitable for both classroom use and self-study. We begin by introducing concentration inequalities along with the notions of covering and packing in metric spaces, and the associated concept of metric entropy. These tools are essential for our analysis. We then introduce the learning-theoretic framework and derive upper bounds on generalization error in terms of metric entropy, Rademacher complexity, and the VC dimension, as well as mutual information and relative entropy. Finally we discuss the minimax estimation framework and establish lower bounds on minimax risk using Fano's inequality, yielding bounds in terms of relative entropy and covering and packing numbers. This manuscript contains preprint of a chapter under consideration for inclusion in the forthcoming third edition of Cover and Thomas's Elements of Information Theory, posted with permission from Wiley. It would follow the chapter posted at arXiv:2605.02989 . The table of contents of the new edition can be found at: https://docs.google.com/document/d/1L-m4oQEJw1PJhoxBeMwrrBD8S_HmvzMEkPbYvS24980/edit?usp=sharing . For feedback, please contact abbas@ee.stanford.edu.