Statistical properties of the site-frequency spectrum associated with Lambda-coalescents

TL;DR

This paper studies the statistical properties of the site frequency spectrum under Lambda-coalescents, deriving recursions for key moments, and develops methods for estimating coalescent parameters from genetic data, distinguishing between different coalescent models.

Contribution

It extends previous results by deriving recursions for moments of the spectrum and introduces a pseudo-likelihood method for parameter estimation in Lambda-coalescents.

Findings

Recursions for expected value, variance, and covariance of the spectrum.

Effective pseudo-likelihood inference for coalescent parameters.

Ability to distinguish Lambda-coalescent subclasses from Kingman coalescent.

Abstract

Statistical properties of the site frequency spectrum associated with Lambda-coalescents are our objects of study. In particular, we derive recursions for the expected value, variance, and covariance of the spectrum, extending earlier results of Fu (1995) for the classical Kingman coalescent. Estimating coalescent parameters introduced by certain Lambda-coalescents for datasets too large for full likelihood methods is our focus. The recursions for the expected values we obtain can be used to find the parameter values which give the best fit to the observed frequency spectrum. The expected values are also used to approximate the probability a (derived) mutation arises on a branch subtending a given number of leaves (DNA sequences), allowing us to apply a pseudo-likelihood inference to estimate coalescence parameters associated with certain subclasses of Lambda coalescents. The properties of the pseudo-likelihood approach are investigated on simulated as well as real mtDNA datasets for the high fecundity Atlantic cod (\emph{Gadus morhua}). Our results for two subclasses of Lambda coalescents show that one can distinguish these subclasses from the Kingman coalescent, as well as between the Lambda-subclasses, even for moderate sample sizes.