Toward a statistical mechanics of four letter words

TL;DR

This paper models four-letter English words as a network of interacting letters using maximum entropy principles, revealing that pairwise correlations capture most of the statistical structure and uncovering insights into word formation.

Contribution

It introduces a maximum entropy framework for modeling letter interactions in four-letter words, effectively capturing their statistical properties and uncovering the structure of English words.

Findings

Maximum entropy models explain ~92% of letter correlations.

Local minima in the energy landscape account for nearly two-thirds of English words.

The model can discover real words not in the training data.

Abstract

We consider words as a network of interacting letters, and approximate the probability distribution of states taken on by this network. Despite the intuition that the rules of English spelling are highly combinatorial (and arbitrary), we find that maximum entropy models consistent with pairwise correlations among letters provide a surprisingly good approximation to the full statistics of four letter words, capturing ~92% of the multi-information among letters and even "discovering" real words that were not represented in the data from which the pairwise correlations were estimated. The maximum entropy model defines an energy landscape on the space of possible words, and local minima in this landscape account for nearly two-thirds of words used in written English.