Symmetry in Shannon's Noiseless Coding Theorem

TL;DR

This paper clarifies and extends Shannon's Noiseless Coding Theorem by introducing a new symmetric formulation, clarifying its domain of applicability, and relating information entropy to physical entropy, thus bridging information theory and physics.

Contribution

It introduces a new symmetric restatement of Shannon's theorem, clarifies its applicability, and explores the connection between information entropy and physical entropy.

Findings

Extended upper bound is achievable, providing symmetry to the theorem.

Clarification of the theorem's domain of application.

Illustration of the relation between information entropy and physical entropy.

Abstract

Statements of Shannon's Noiseless Coding Theorem by various authors, including the original, are reviewed and clarified. Traditional statements of the theorem are often unclear as to when it applies. A new notation is introduced and the domain of application is clarified. An examination of the bounds of the Theorem leads to a new symmetric restatement. It is shown that the extended upper bound is an acheivable upper bound, giving symmetry to the theorem.The relation of information entropy to the physical entropy of Gibbs and Boltmann is illustrated. Consequently, the study of Shannon Entropy is strongly related to physics and there is a physical theory of information. This paper is the beginning of of an attempt to clarify these relationships.