An intuition for physicists: information gain from experiments

TL;DR

This paper explains the concept of information gain in experiments, specifically how one bit of information corresponds to a Gaussian distribution's standard deviation shrinking by a factor of three, providing intuitive understanding for physicists.

Contribution

It offers an intuitive interpretation of information gain in experiments, linking it to the quantitative change in Gaussian distributions.

Findings

One bit of information reduces Gaussian standard deviation by a factor of three.

Provides a clear, intuitive understanding of Kullback-Leibler divergence in experimental contexts.

Connects information theory with practical implications in physics experiments.

Abstract

How much one has learned from an experiment is quantifiable by the information gain, also known as the Kullback-Leibler divergence. The narrowing of the posterior parameter distribution $P(\theta|D)$ compared with the prior parameter distribution $\pi(\theta)$, is quantified in units of bits, as: $ D_{\mathrm{KL}}(P|\pi)=\int\log_{2}\left(\frac{P(\theta|D)}{\pi(\theta)}\right)\,P(\theta|D)\,d\theta $. This research note gives an intuition what one bit of information gain means. It corresponds to a Gaussian shrinking its standard deviation by a factor of three.