Laws of thermodynamics for exponential families

TL;DR

This paper develops thermodynamic laws within exponential families, translating concepts like work, heat, and cycles into AI and statistical learning contexts, with implications for distribution shift analysis.

Contribution

It extends thermodynamic principles to exponential families, bridging statistical mechanics and learning, and introduces new interpretations of energy and equilibrium in AI.

Findings

Thermodynamic concepts are mapped to learning scenarios.

New insights into distribution shift quantification.

Exact counterparts of thermodynamic cycles in AI.

Abstract

We develop the laws of thermodynamics in terms of general exponential families. By casting learning (log-loss minimization) problems in max-entropy and statistical mechanics terms, we translate thermodynamics results to learning scenarios. We extend the well-known way in which exponential families characterize thermodynamic and learning equilibria. Basic ideas of work and heat, and advanced concepts of thermodynamic cycles and equipartition of energy, find exact and useful counterparts in AI / statistics terms. These ideas have broad implications for quantifying and addressing distribution shift.