Bias in Zipf's Law Estimators

TL;DR

This paper examines biases in Zipf's law estimators, derives the correct likelihood function, explores an approximate Bayesian computation method, and discusses the limitations of these methods when applied to natural language data.

Contribution

It derives the correct likelihood function for Zipf's law, evaluates an ABC estimator, and highlights biases in existing methods when applied to complex language data.

Findings

MLE estimators are biased due to incorrect likelihood functions

ABC method reduces bias for idealized Zipfian data

Bias persists in natural language applications due to complex data structure

Abstract

The prevailing maximum likelihood estimators for inferring power law models from rank-frequency data are biased. The source of this bias is an inappropriate likelihood function. The correct likelihood function is derived and shown to be computationally intractable. A more computationally efficient method of approximate Bayesian computation (ABC) is explored. This method is shown to have less bias for data generated from idealised rank-frequency Zipfian distributions. However, the existing estimators and the ABC estimator described here assume that words are drawn from a simple probability distribution, while language is a much more complex process. We show that this false assumption leads to continued biases when applying any of these methods to natural language to estimate Zipf exponents. We recommend that researchers be aware of these biases when investigating power laws in rank-frequency data.