Shannon's entropy revisited

TL;DR

This paper revisits Shannon's entropy, proposing a new formula that accounts for finite sample sizes, with implications for communication efficiency and protocol design.

Contribution

It introduces a finite-sample entropy formula satisfying Shannon's axioms, bridging the gap between theoretical and practical entropy calculations.

Findings

Proposed a new entropy formula for finite samples

Analyzed the physical meaning of entropy differences

Discussed practical implications for communication systems

Abstract

I consider the effect of a finite sample size on the entropy of a sample of independent events. I propose formula for entropy which satisfies Shannon's axioms, and which reduces to Shannon's entropy when sample size is infinite. I discuss the physical meaning of the difference between two formulas, including some practical implications, such as maximum achievable channel utilization, and minimum achievable communication protocol overhead, for a given message size.