Lectures on Probability, Entropy, and Statistical Physics

TL;DR

This paper explores the foundations of inductive inference, probability, and entropy, aiming to develop Bayesian tools for reasoning under uncertainty and illustrating their application in classical statistical physics.

Contribution

It provides a comprehensive Bayesian framework for understanding probability and entropy, connecting information theory with statistical physics.

Findings

Develops Bayesian methods for inductive inference

Clarifies the role of entropy in information processing

Applies concepts to classical statistical physics

Abstract

These lectures deal with the problem of inductive inference, that is, the problem of reasoning under conditions of incomplete information. Is there a general method for handling uncertainty? Or, at least, are there rules that could in principle be followed by an ideally rational mind when discussing scientific matters? What makes one statement more plausible than another? How much more plausible? And then, when new information is acquired how do we change our minds? Or, to put it differently, are there rules for learning? Are there rules for processing information that are objective and consistent? Are they unique? And, come to think of it, what, after all, is information? It is clear that data contains or conveys information, but what does this precisely mean? Can information be conveyed in other ways? Is information physical? Can we measure amounts of information? Do we need to? Our goal is to develop the main tools for inductive inference--probability and entropy--from a thoroughly Bayesian point of view and to illustrate their use in physics with examples borrowed from the foundations of classical statistical physics.