Physics of the Shannon Limits

TL;DR

This paper offers a physical interpretation of fundamental information theory inequalities by linking them to the second law of thermodynamics, suggesting that the limits of information theory are rooted in physical laws.

Contribution

It introduces a thermodynamic perspective on the information inequality, connecting the foundations of Shannon theory to physical principles like the second law of thermodynamics.

Findings

The information inequality can be interpreted through thermodynamics.

Fundamental limits of information theory are rooted in physical laws.

The second law of thermodynamics underpins key information-theoretic principles.

Abstract

We provide a simple physical interpretation, in the context of the second law of thermodynamics, to the information inequality (a.k.a. the Gibbs' inequality, which is also equivalent to the log-sum inequality), asserting that the relative entropy between two probability distributions cannot be negative. Since this inequality stands at the basis of the data processing theorem (DPT), and the DPT in turn is at the heart of most, if not all, proofs of converse theorems in Shannon theory, it is observed that conceptually, the roots of fundamental limits of Information Theory can actually be attributed to the laws of physics, in particular, to the second law of thermodynamics, and at least indirectly, also to the law of energy conservation. By the same token, in the other direction: one can view the second law as stemming from information-theoretic principles.