A statistical mechanical interpretation of instantaneous codes

TL;DR

This paper offers a novel interpretation of noiseless source coding through statistical mechanics, linking concepts like entropy and temperature to code properties, and revealing that temperature 1 equals the average codeword length.

Contribution

It introduces a new statistical mechanical framework for understanding instantaneous codes, connecting thermodynamic concepts to information theory.

Findings

Temperature 1 equals average codeword length

Statistical mechanical entropy relates to source coding entropy

Thermodynamic analogies provide intuitive derivations of information-theoretic relations

Abstract

In this paper we develop a statistical mechanical interpretation of the noiseless source coding scheme based on an absolutely optimal instantaneous code. The notions in statistical mechanics such as statistical mechanical entropy, temperature, and thermal equilibrium are translated into the context of noiseless source coding. Especially, it is discovered that the temperature 1 corresponds to the average codeword length of an instantaneous code in this statistical mechanical interpretation of noiseless source coding scheme. This correspondence is also verified by the investigation using box-counting dimension. Using the notion of temperature and statistical mechanical arguments, some information-theoretic relations can be derived in the manner which appeals to intuition.