Information Theory and the Length Distribution of all Discrete Systems

TL;DR

This paper reveals that diverse discrete systems such as proteins and computer functions share similar length distributions, explained by a new information-theoretic framework based on the Conservation of Hartley-Shannon information, which predicts universal patterns like power-laws and Zipf's law.

Contribution

It introduces a novel theoretical framework embedding CoHSI into statistical mechanics, unifying the length distributions of heterogeneous and homogeneous discrete systems and validating predictions across multiple domains.

Findings

Length distributions of proteins and computer functions are qualitatively identical.

The CoHSI framework predicts unimodal and power-law distributions for different system types.

Different but related alphabets in heterogeneous systems follow power-law relations.

Abstract

We begin with the extraordinary observation that the length distribution of 80 million proteins in UniProt, the Universal Protein Resource, measured in amino acids, is qualitatively identical to the length distribution of large collections of computer functions measured in programming language tokens, at all scales. That two such disparate discrete systems share important structural properties suggests that yet other apparently unrelated discrete systems might share the same properties, and certainly invites an explanation. We demonstrate that this is inevitable for all discrete systems of components built from tokens or symbols. Departing from existing work by embedding the Conservation of Hartley-Shannon information (CoHSI) in a classical statistical mechanics framework, we identify two kinds of discrete system, heterogeneous and homogeneous. Heterogeneous systems contain components built from a unique alphabet of tokens and yield an implicit CoHSI distribution with a sharp unimodal peak asymptoting to a power-law. Homogeneous systems contain components each built from just one kind of token unique to that component and yield a CoHSI distribution corresponding to Zipf's law. This theory is applied to heterogeneous systems, (proteome, computer software, music); homogeneous systems (language texts, abundance of the elements); and to systems in which both heterogeneous and homogeneous behaviour co-exist (word frequencies and word length frequencies in language texts). In each case, the predictions of the theory are tested and supported to high levels of statistical significance. We also show that in the same heterogeneous system, different but consistent alphabets must be related by a power-law. We demonstrate this on a large body of music by excluding and including note duration in the definition of the unique alphabet of notes.