Carnot in the Information Age: Discrete Symmetric Channels

TL;DR

This paper applies a thermodynamic framework to analyze the mutual information of symmetric discrete channels, deriving explicit formulas and revealing a universal simple form for channels with equiprobable inputs and outputs.

Contribution

It introduces a thermodynamic approach to compute mutual information in symmetric channels, generalizing the second law for temperature-dependent energy levels and deriving a universal formula.

Findings

Explicit mutual information formulas for binary and 4-symbol symmetric channels

A universal simple form of mutual information for equiprobable I/O channels

Validation of thermodynamic principles in information theory context

Abstract

Modeling communication channels as thermal systems results in Hamiltonians which are an explicit function of the temperature. The first two authors have recently generalized the second thermodynamic law to encompass systems with temperature-dependent energy levels, $dQ=TdS+<d\mathcal{E}/dT>dT$, where {$<\cdot>$} denotes averaging over the Boltzmann distribution, recomputing the mutual information and other main properties of the popular Gaussian channel. Here the mutual information for the binary symmetric channel as well as for the discrete symmetric channel consisting of 4 input/output (I/O) symbols is explicitly calculated using the generalized second law of thermodynamics. For equiprobable I/O the mutual information of the examined channels has a very simple form, -$\gamma U(\gamma)|_0^\beta$, where $U$ denotes the internal energy of the channel. We prove that this simple form of the mutual information governs the class of discrete memoryless symmetric communication channels with equiprobable I/O symbols.