The Structure of Genealogies in the Presence of Purifying Selection: A "Fitness-Class Coalescent"

TL;DR

This paper introduces a 'fitness-class coalescent' model to better understand how purifying selection influences genealogical structures and genetic variation patterns at linked sites, extending classical coalescent theory.

Contribution

The paper develops an effective coalescent framework that incorporates purifying selection at multiple linked sites, bridging previous models and enabling new analytical insights.

Findings

Purifying selection distorts genealogical structures compared to neutral models.

The model predicts specific patterns of genetic variation at selected and linked neutral sites.

Selection effects resemble population size changes in neutral models.

Abstract

Compared to a neutral model, purifying selection distorts the structure of genealogies and hence alters the patterns of sampled genetic variation. Although these distortions may be common in nature, our understanding of how we expect purifying selection to affect patterns of molecular variation remains incomplete. Genealogical approaches such as coalescent theory have proven difficult to generalize to situations involving selection at many linked sites, unless selection pressures are extremely strong. Here, we introduce an effective coalescent theory (a "fitness-class coalescent") to describe the structure of genealogies in the presence of purifying selection at many linked sites. We use this effective theory to calculate several simple statistics describing the expected patterns of variation in sequence data, both at the sites under selection and at linked neutral sites. Our analysis combines our earlier description of the allele frequency spectrum in the presence of purifying selection (Desai et al. 2010) with the structured coalescent approach of Nordborg (1997), to trace the ancestry of individuals through the distribution of fitnesses within the population. Alternatively, we can derive our results using an extension of the coalescent approach of Hudson and Kaplan (1994). We find that purifying selection leads to patterns of genetic variation that are related but not identical to a neutrally evolving population in which population size has varied in a specific way in the past.