Quadri-allele frequency spectrum in a coalescent topology for mutations in non-constant population size

TL;DR

This paper develops a model for the quadri-allele frequency spectrum in a coalescent framework, accounting for non-constant population sizes and divergence into three populations, to better understand genetic variation.

Contribution

It introduces a novel analytical model for quadri-allele frequency spectrum considering non-constant population size and divergence without migration.

Findings

Model accurately describes allele frequency distributions.

Applicable to populations with complex demographic histories.

Provides insights into mutation patterns in diverged populations.

Abstract

The sample frequency spectrum of a segregating site is the probability distribution of a sample of alleles from a genetic locus, conditional on observing the sample to have more than one clearly different phenotypes. We present a model for analyzing quadri-allele frequency spectrum, where the ancestral population diverged into three populations at a certain divergence time and the resulting mutations on the branches of the coalescent tree gave rise to three different derived alleles, which could be observed in the present generation along with the ancestral allele. The model has been analyzed for non-constant population size, assuming we had a certain number of extant lineages at the divergence time and no migration occurs between the populations.