A note on the expected minimum error probability in equientropic channels

TL;DR

This paper investigates the relationship between code quality and error probability in equientropic channels, showing that codes with maximal marginal entropy minimize expected error, exemplified by AWGN channels.

Contribution

It characterizes the upper bound on expected minimum error probability for codes in equientropic channels and links maximal marginal entropy to optimal code performance.

Findings

Maximal marginal entropy minimizes expected error in equientropic channels.

Random coding maximizes marginal entropy in AWGN channels.

Theoretical link between entropy and error probability in finite message regimes.

Abstract

While the channel capacity reflects a theoretical upper bound on the achievable information transmission rate in the limit of infinitely many bits, it does not characterise the information transfer of a given encoding routine with finitely many bits. In this note, we characterise the quality of a code (i. e. a given encoding routine) by an upper bound on the expected minimum error probability that can be achieved when using this code. We show that for equientropic channels this upper bound is minimal for codes with maximal marginal entropy. As an instructive example we show for the additive white Gaussian noise (AWGN) channel that random coding---also a capacity achieving code---indeed maximises the marginal entropy in the limit of infinite messages.