Quenched coalescent for diploid population models with selfing and overlapping generations

TL;DR

This paper develops a quenched coalescent model for diploid populations with selfing and overlapping generations, revealing how pedigree influences gene genealogies and the site-frequency spectrum in high selfing regimes.

Contribution

It introduces a novel quenched coalescent framework conditioned on pedigrees, extending traditional models to include selfing and overlapping generations.

Findings

Conditional gene genealogy converges to a random measure influenced by pedigree.

The model generalizes coalescing random walks on ancestral recombination graphs.

Provides insights into how pedigree affects the site-frequency spectrum.

Abstract

We introduce a general diploid population model with self-fertilization and possible overlapping generations, and study the genealogy of a sample of $n$ genes as the population size $N$ tends to infinity. Unlike traditional approach in coalescent theory which considers the unconditional (annealed) law of the gene genealogies averaged over the population pedigree, here we study the conditional (quenched) law of gene genealogies given the pedigree. We focus on the case of high selfing probability and obtain that this conditional law converges to a random probability measure, given by the random law of a system of coalescing random walks on an exchangeable fragmentation-coalescence process of \cite{berestycki04}. This system contains the system of coalescing random walks on the ancestral recombination graph as a special case, and it sheds new light on the site-frequency spectrum (SFS) of genetic data by specifying how SFS depends on the pedigree. The convergence result is proved by means of a general characterization of weak convergence for random measures on the Skorokhod space with paths taking values in a locally compact Polish space.