Information Theory and Statistical Learning

TL;DR

This chapter explores the intersection of information theory and statistical learning, focusing on divergence measures like ELBO, f-divergences, and Fisher divergence, with applications to various models including diffusion and GANs.

Contribution

It offers a concise, accessible overview of divergence measures in model training, with a systematic derivation for diffusion models, suitable for educational purposes.

Findings

Systematic derivation of diffusion model training

Introduction of ELBO, f-divergences, and Fisher divergence

Application examples include GANs and autoencoders

Abstract

This manuscript contains preprint of a chapter under consideration for inclusion in the forthcoming third edition of {\em Cover and Thomas's Elements of Information Theory}, posted with permission from Wiley. The table of contents EIT-3 ToC of the new edition can be found at: https://docs.google.com/document/d/1L-m4oQEJw1PJhoxBeMwrrBD8S_HmvzMEkPbYvS24980/edit?usp=sharing . For feedback, please contact abbas@ee.stanford.edu Learning and information theory intersect in both model training and the characterization of fundamental performance limits. This manuscript provides a concise and accessible treatment of the first intersection, requiring only basic background in information theory and statistics at the senior undergraduate or first-year graduate level. End-of-chapter exercises make the material well suited for classroom use as well as self-study. The chapter focuses on the role of divergence measures in model training, with examples ranging from linear and logistic regression to autoregressive models, variational autoencoders, diffusion models, generative adversarial networks, and score-based models. It introduces the evidence lower bound (ELBO), $f$\!-divergences, and the Fisher divergence. In particular, the treatment of the generative diffusion model provides a more systematic and explicit derivation than is typical in the literature.