A conditional likelihood is required to estimate the selection coefficient in ancient DNA

TL;DR

This paper introduces a conditional likelihood approach for accurately estimating the selection coefficient from limited ancient DNA allele frequency trajectories, addressing biases of traditional methods.

Contribution

It develops a new likelihood function that accounts for the conditioning of single allele frequency trajectories, improving estimation accuracy in ancient DNA studies.

Findings

Conditional likelihood provides precise estimates near fixation.

Unconditioned likelihood often yields unfalsifiable or incorrect results.

The method improves inference from limited, single trajectories.

Abstract

Time-series of allele frequencies are a useful and unique set of data to determine the strength of natural selection on the background of genetic drift. Technically, the selection coefficient is estimated by means of a likelihood function built under the hypothesis that the available trajectory spans a sufficiently large portion of the fitness landscape. Especially for ancient DNA, however, often only one single such trajectories is available and the coverage of the fitness landscape is very limited. In fact, one single trajectory is more representative of a process conditioned both in the initial and in the final condition than of a process free to visit the available fitness landscape. Based on two models of population genetics, here we show how to build a likelihood function for the selection coefficient that takes the statistical peculiarity of single trajectories into account. We show that this conditional likelihood delivers a precise estimate of the selection coefficient also when allele frequencies are close to fixation whereas the unconditioned likelihood fails. Finally, we discuss the fact that the traditional, unconditioned likelihood always delivers an answer, which is often unfalsifiable and appears reasonable also when it is not correct.