Learning to explore tree neighbourhoods for phylogenetic inference

TL;DR

This paper introduces a reinforcement learning approach to solve the phylogenetic inference problem more efficiently and effectively.

Contribution

A novel reinforcement learning framework for the balanced minimum evolution problem in phylogenetics is proposed.

Findings

The RL agent solves instances with up to 100 taxa and outperforms greedy heuristics.

The method matches state-of-the-art algorithms and adapts well to distributional shifts.

A search-based framework improves performance during evaluation.

Abstract

Phylogenetic inference is a key challenge in computational biology, with applications ranging from evolutionary analysis to comparative genomics. The balanced minimum evolution problem (BMEP) offers a well-established formulation of this problem, but remains computationally intractable for large instances. In this work, we propose a reinforcement learning (RL) framework to tackle the BMEP through local search in the space of phylogenetic trees. Our contributions are three-fold: (i) we introduce an improved RL formulation tailored to the structure of phylogenetic inference in the context of the BMEP; (ii) we train an RL agent capable of solving instances with up to 100 taxa; and (iii) we investigate the generalization capabilities of the learned policy across different substitution models, instance sizes, and datasets. To address the limitations of relying solely on the learned policy at inference time, we integrate it into a novel search-based framework that enables effective adaptation during evaluation. Experimental results show that our method outperforms greedy heuristics and matches the performance of state-of-the-art algorithms for the BMEP. When tested under significant distributional shifts, we greatly reduce the gap with state-of-the-art algorithms. This demonstrates the potential of RL applications to phylogenetic inference.

Figures