A Two Parameters Equation for Word Rank-Frequency Relation

TL;DR

This paper proposes a two-parameter mathematical model to accurately fit the rank-frequency relation of words in language data, improving understanding of linguistic patterns.

Contribution

It introduces a novel two-parameter equation for modeling word rank-frequency relations, extending previous models with better fit and parameter interpretability.

Findings

The model fits well with well-behaved linguistic data.

Parameters s and t are estimated from data with specific constraints.

The model generalizes previous rank-frequency relations.

Abstract

Let $f (\cdot)$ be the absolute frequency of words and $r$ be the rank of words in decreasing order of frequency, then the following function can fit the rank-frequency relation \[ f (r;s,t) = \left(\frac{r_{\tt max}}{r}\right)^{1-s} \left(\frac{r_{\tt max}+t \cdot r_{\tt exp}}{r+t \cdot r_{\tt exp}}\right)^{1+(1+t)s} \] where $r_{\tt max}$ and $r_{\tt exp}$ are the maximum and the expectation of the rank, respectively; $s>0$ and $t>0$ are parameters estimated from data. On well-behaved data, there should be $s<1$ and $s \cdot t < 1$.