On the Computability of Finding Capacity-Achieving Codes

TL;DR

This paper proves that there exists a Turing machine capable of constructing capacity-achieving codes for discrete memoryless channels, given computable channel parameters, by systematically searching for valid codes within the theoretical framework of recursive functions.

Contribution

It formalizes the construction of capacity-achieving codes as a computable process using recursive functions, extending Shannon's theorem into the realm of algorithmic computability.

Findings

Existence of a Turing machine that constructs capacity-achieving codes.

Formalization of code construction as a μ-recursive function.

Extension to cases with computable error tolerance and rate.

Abstract

This work studies the problem of constructing capacity-achieving codes from an algorithmic perspective. Specifically, we prove that there exists a Turing machine which, given a discrete memoryless channel $p_{Y|X}$, a target rate $R$ less than the channel capacity $C(p_{Y|X})$, and an error tolerance $\epsilon > 0$, outputs a block code $\mathcal{C}$ achieving a rate at least $R$ and a maximum block error probability below $\epsilon$. The machine operates in the general case where all transition probabilities of $p_{Y|X}$ are computable real numbers, and the parameters $R$ and $\epsilon$ are rational. The proof builds on Shannon's channel coding theorem and relies on an exhaustive search approach that systematically enumerates all codes of increasing block length until a valid code is found. This construction is formalized using the theory of recursive functions, yielding a $\mu$-recursive function $\mathrm{FindCode} : \mathbb{N}^3 \rightharpoonup \mathbb{N}$ that takes as input appropriate encodings of $p_{Y|X}$, $R$, and $\epsilon$, and, whenever $R < C(p_{Y|X})$, outputs an encoding of a valid code. By Kleene's normal form theorem, which establishes the computational equivalence between Turing machines and $\mu$-recursive functions, we conclude that the problem is solvable by a Turing machine. This result can also be extended to the case where $\epsilon$ is a computable real number, while we further discuss an analogous generalization of our analysis when $R$ is computable as well. We note that the assumptions that the probabilities of $p_{Y|X}$, as well as $\epsilon$ and $R$, are computable real numbers cannot be further weakened, since computable reals constitute the largest subset of $\mathbb{R}$ representable by algorithmic means.