The Epic Story of Maximum Likelihood

TL;DR

This paper reviews the historical development of maximum likelihood estimation, highlighting Fisher's contributions, the mathematical foundations, and the evolution of ideas from early thinkers to modern statisticians.

Contribution

It uncovers the historical progression of maximum likelihood, emphasizing Fisher's unpublished conditions for estimator consistency and efficiency, and links to classical mathematical principles.

Findings

Fisher's 1930 conditions for MLE consistency and efficiency are detailed.

The derivation of the information inequality from analysis of variance is explained.

Fisher's approach via estimating functions is connected to Euler's relation for homogeneous functions.

Abstract

At a superficial level, the idea of maximum likelihood must be prehistoric: early hunters and gatherers may not have used the words ``method of maximum likelihood'' to describe their choice of where and how to hunt and gather, but it is hard to believe they would have been surprised if their method had been described in those terms. It seems a simple, even unassailable idea: Who would rise to argue in favor of a method of minimum likelihood, or even mediocre likelihood? And yet the mathematical history of the topic shows this ``simple idea'' is really anything but simple. Joseph Louis Lagrange, Daniel Bernoulli, Leonard Euler, Pierre Simon Laplace and Carl Friedrich Gauss are only some of those who explored the topic, not always in ways we would sanction today. In this article, that history is reviewed from back well before Fisher to the time of Lucien Le Cam's dissertation. In the process Fisher's unpublished 1930 characterization of conditions for the consistency and efficiency of maximum likelihood estimates is presented, and the mathematical basis of his three proofs discussed. In particular, Fisher's derivation of the information inequality is seen to be derived from his work on the analysis of variance, and his later approach via estimating functions was derived from Euler's Relation for homogeneous functions. The reaction to Fisher's work is reviewed, and some lessons drawn.