Fixation probability of rare nonmutator and evolution of mutation rates

TL;DR

This paper investigates how selection and genetic drift influence the evolution of mutation rates in asexual populations, revealing complex dynamics of fixation probabilities and a novel mutation rate-population size relationship.

Contribution

It introduces a multitype branching process model to analyze the fixation probability of nonmutators and uncovers nonmonotonic effects of compensatory mutations under different selection strengths.

Findings

Nonmutators are more likely to fix when mutator deleterious mutation rates are high.

Compensatory mutations decrease fixation probability under strong selection.

Fixation probability varies nonmonotonically with compensatory mutation rate under mild selection.

Abstract

Although mutations drive the evolutionary process, the rates at which the mutations occur are themselves subject to evolutionary forces. Our purpose here is to understand the role of selection and random genetic drift in the evolution of mutation rates, and we address this question in asexual populations at mutation-selection equilibrium neglecting selective sweeps. Using a multitype branching process, we calculate the fixation probability of a rare nonmutator in a large asexual population of mutators, and find that a nonmutator is more likely to fix when the deleterious mutation rate of the mutator population is high. Compensatory mutations in the mutator population are found to decrease the fixation probability of a nonmutator when the selection coefficient is large. But, surprisingly, the fixation probability changes nonmonotonically with increasing compensatory mutation rate when the selection is mild. Using these results for the fixation probability and a drift-barrier argument, we find a novel relationship between the mutation rates and the population size. We also discuss the time to fix the nonmutator in an adapted population of asexual mutators, and compare our results with experiments.