Word forms - not just their lengths- are optimized for efficient communication

TL;DR

This paper shows that word forms are optimized for efficient communication not just by length but through their distinctiveness, which better predicts word frequency across multiple languages, reflecting processing constraints.

Contribution

It introduces phonological information content as a measure of word distinctiveness and demonstrates its superiority over length in predicting word frequency across 13 languages.

Findings

Distinctiveness outperforms length in predicting frequency

Phonological information content captures word distinctiveness effectively

Cross-linguistic evidence supports processing constraints shaping word forms

Abstract

The inverse relationship between the length of a word and the frequency of its use, first identified by G.K. Zipf in 1935, is a classic empirical law that holds across a wide range of human languages. We demonstrate that length is one aspect of a much more general property of words: how distinctive they are with respect to other words in a language. Distinctiveness plays a critical role in recognizing words in fluent speech, in that it reflects the strength of potential competitors when selecting the best candidate for an ambiguous signal. Phonological information content, a measure of a word's string probability under a statistical model of a language's sound or character sequences, concisely captures distinctiveness. Examining large-scale corpora from 13 languages, we find that distinctiveness significantly outperforms word length as a predictor of frequency. This finding provides evidence that listeners' processing constraints shape fine-grained aspects of word forms across languages.