Fitting Ranked English and Spanish Letter Frequency Distribution in U.S. and Mexican Presidential Speeches

TL;DR

This study compares how well ten different functions fit letter frequency distributions in U.S. and Mexican presidential speeches over two centuries, finding the Cocho/Beta function as the best fit and revealing stable letter usage.

Contribution

It introduces a comprehensive comparison of multiple functions for modeling letter frequency distributions and applies a novel clustering method based on observed-over-expected ratios.

Findings

The Cocho/Beta function best fits the data according to model selection criteria.

Letter usage remains generally stable over time despite some ranking shifts.

A new clustering method identifies groups of letters by frequency ratios.

Abstract

The limited range in its abscissa of ranked letter frequency distributions causes multiple functions to fit the observed distribution reasonably well. In order to critically compare various functions, we apply the statistical model selections on ten functions, using the texts of U.S. and Mexican presidential speeches in the last 1-2 centuries. Dispite minor switching of ranking order of certain letters during the temporal evolution for both datasets, the letter usage is generally stable. The best fitting function, judged by either least-square-error or by AIC/BIC model selection, is the Cocho/Beta function. We also use a novel method to discover clusters of letters by their observed-over-expected frequency ratios.