On the importance of being structured: instantaneous coalescence rates and a re-evaluation of human evolution

TL;DR

This paper demonstrates that ignoring population structure in genetic inference methods leads to misleading conclusions, introduces the IICR parameter to interpret coalescence data, and re-evaluates human evolutionary history.

Contribution

The authors develop the IICR parameter to account for population structure and show its implications for demographic inference, challenging assumptions of panmixia in population genetics.

Findings

Population structure can produce false signals of population size change.

The IICR parameter accurately describes coalescence in structured populations.

Re-analysis of human data reveals historical changes in gene flow and population connectivity.

Abstract

Most species are structured and influenced by processes that either increased or reduced gene flow between populations. However, most population genetic inference methods ignore population structure and reconstruct a history characterized by population size changes under the assumption that species behave as panmictic units. This is potentially problematic since population structure can generate spurious signals of population size change. Moreover, when the model assumed for demographic inference is misspecified, genomic data will likely increase the precision of misleading if not meaningless parameters. In a context of model uncertainty (panmixia \textit{versus} structure) genomic data may thus not necessarily lead to improved statistical inference. We consider two haploid genomes and develop a theory which explains why any demographic model (with or without population size changes) will necessarily be interpreted as a series of changes in population size by inference methods ignoring structure. We introduce a new parameter, the IICR (inverse instantaneous coalescence rate), and show that it is equivalent to a population size only in panmictic models, and mostly misleading for structured models. We argue that this general issue affects all population genetics methods ignoring population structure. We take the PSMC method as an example and show that it infers population size changes that never took place. We apply our approach to human genomic data and find a reduction in gene flow at the start of the Pleistocene, a major increase throughout the Middle-Pleistocene, and an abrupt disconnection preceding the emergence of modern humans.