Normalizing Kernels in the Billera-Holmes-Vogtmann Treespace

TL;DR

This paper introduces an improved kernel density estimation method for phylogenetic tree space, utilizing holonomic gradient techniques to normalize kernels, which enhances detection of outlier gene trees and evolutionary events.

Contribution

It presents a novel approach to normalize kernels in the Billera-Holmes-Vogtmann treespace using holonomic gradient methods, improving outlier detection in phylogenetic analyses.

Findings

Identified unreliable sequence alignments missed by previous methods

Detected a gene with potential horizontal gene transfer

Enhanced outlier detection in phylogenetic tree analysis

Abstract

As costs of genome sequencing have dropped precipitously, development of efficient bioinformatic methods to analyze genome structure and evolution have become ever more urgent. For example, most published phylogenomic studies involve either massive concatenation of sequences, or informal comparisons of phylogenies inferred on a small subset of orthologous genes, neither of which provides a comprehensive overview of evolution or systematic identification of genes with unusual and interesting evolution (e.g. horizontal gene transfers, gene duplication and subsequent neofunctionalization). We are interested in identifying such "outlying" gene trees from the set of gene trees and estimating the distribution of the tree over the "tree space". This paper describes an improvement to the kdetrees algorithm, an adaptation of classical kernel density estimation to the metric space of phylogenetic trees (Billera-Holmes-Vogtman treespace), whereby the kernel normalizing constants, are estimated through the use of the novel holonomic gradient methods. As the original kdetrees paper, we have applied kdetrees to a set of Apicomplexa genes and it identified several unreliable sequence alignments which had escaped previous detection, as well as a gene independently reported as a possible case of horizontal gene transfer.