Asymptotic properties of the number of matching coalescent histories for caterpillar-like families of species trees

TL;DR

This paper investigates the asymptotic growth of the number of matching coalescent histories in caterpillar-like species trees, revealing they grow proportionally to Catalan numbers with a constant depending on the seed tree.

Contribution

It extends previous results to arbitrary seed trees, providing a generating function approach and a method to compute the asymptotic constant for coalescent histories.

Findings

Number of coalescent histories grows asymptotically as a constant times Catalan numbers.

The asymptotic constant depends on the seed tree shape.

A procedure to compute the constant and generating function from the seed tree is introduced.

Abstract

Coalescent histories provide lists of species tree branches on which gene tree coalescences can take place, and their enumerative properties assist in understanding the computational complexity of calculations central in the study of gene trees and species trees. Here, we solve an enumerative problem left open by Rosenberg (IEEE/ACM Transactions on Computational Biology and Bioinformatics 10: 1253-1262, 2013) concerning the number of coalescent histories for gene trees and species trees with a matching labeled topology that belongs to a generic caterpillar-like family. By bringing a generating function approach to the study of coalescent histories, we prove that for any caterpillar-like family with seed tree $t$, the sequence $(h_n)_{n\geq 0}$ describing the number of matching coalescent histories of the $n$th tree of the family grows asymptotically as a constant multiple of the Catalan numbers. Thus, $h_n \sim \beta_t c_n$, where the asymptotic constant $\beta_t > 0$ depends on the shape of the seed tree $t$. The result extends a claim demonstrated only for seed trees with at most 8 taxa to arbitrary seed trees, expanding the set of cases for which detailed enumerative properties of coalescent histories can be determined. We introduce a procedure that computes from $t$ the constant $\beta_t$ as well as the algebraic expression for the generating function of the sequence $(h_n)_{n\geq 0}$.