On the class of coding optimality of human languages and the origins of Zipf's law

TL;DR

This paper introduces a new class of coding optimality that explains Zipf's law in human languages and some animal communication systems, linking frequency distributions to coding efficiency and optimality.

Contribution

It defines a novel class of optimality for coding systems, connecting Zipf's law, size-rank, and size-probability laws, and explores their implications for human and animal communication.

Findings

Human languages fit the new optimality class.

Some animal communication systems exhibit exponential distributions.

Straight lines in log-log plots indicate near-optimal coding.

Abstract

Here we present a new class of optimality for coding systems. Members of that class are displaced linearly from optimal coding and thus exhibit Zipf's law, namely a power-law distribution of frequency ranks. Within that class, Zipf's law, the size-rank law and the size-probability law form a group-like structure. We identify human languages that are members of the class. All languages showing sufficient agreement with Zipf's law are potential members of the class. In contrast, there are communication systems in other species that cannot be members of that class for exhibiting an exponential distribution instead but dolphins and humpback whales might. We provide a new insight into plots of frequency versus rank in double logarithmic scale. For any system, a straight line in that scale indicates that the lengths of optimal codes under non-singular coding and under uniquely decodable encoding are displaced by a linear function whose slope is the exponent of Zipf's law. For systems under compression and constrained to be uniquely decodable, such a straight line may indicate that the system is coding close to optimality. We provide support for the hypothesis that Zipf's law originates from compression and define testable conditions for the emergence of Zipf's law in compressing systems.