Maximum likelihood estimates under $\mathbf{k}$-allele models with selection can be numerically unstable

TL;DR

This paper investigates the numerical instability of maximum likelihood estimates in k-allele Wright-Fisher models with selection and proposes a Bayesian approach for more stable inference.

Contribution

It identifies the unbounded likelihood issue at certain data points and introduces a Bayesian method to improve the stability and accuracy of parameter estimation.

Findings

Likelihood is unbounded at certain data points under selection.

Standard MLE approaches are numerically unstable with substantial selection.

Bayesian posterior provides more reliable interval estimates.

Abstract

The stationary distribution of allele frequencies under a variety of Wright--Fisher $k$-allele models with selection and parent independent mutation is well studied. However, the statistical properties of maximum likelihood estimates of parameters under these models are not well understood. Under each of these models there is a point in data space which carries the strongest possible signal for selection, yet, at this point, the likelihood is unbounded. This result remains valid even if all of the mutation parameters are assumed to be known. Therefore, standard simulation approaches used to approximate the sampling distribution of the maximum likelihood estimate produce numerically unstable results in the presence of substantial selection. We describe the Bayesian alternative where the posterior distribution tends to produce more accurate and reliable interval estimates for the selection intensity at a locus.