PlantBiMoE: A Bidirectional Foundation Model with SparseMoE for Plant Genomes

TL;DR

PlantBiMoE is a novel bidirectional plant genome language model that uses SparseMoE to improve efficiency and performance across diverse genomic tasks, advancing computational biology.

Contribution

It introduces PlantBiMoE, combining bidirectional modeling with SparseMoE for the first time in plant genomics, enhancing efficiency and accuracy.

Findings

Achieves top performance on 20 out of 31 datasets

Reduces parameters significantly with SparseMoE

Effective in diverse genomic tasks

Abstract

Understanding the underlying linguistic rules of plant genomes remains a fundamental challenge in computational biology. Recent advances including AgroNT and PDLLMs have made notable progress although, they suffer from excessive parameter size and limited ability to model the bidirectional nature of DNA strands respectively. To address these limitations, we propose PlantBiMoE, a lightweight and expressive plant genome language model that integrates bidirectional Mamba and a Sparse Mixture-of-Experts (SparseMoE) framework. The bidirectional Mamba enables the model to effectively capture structural dependencies across both the forward and reverse DNA strands, while SparseMoE significantly reduces the number of active parameters, improving computational efficiency without sacrificing modeling capacity. We evaluated and tested our model on the Modified Plants Genome Benchmark (MPGB), an enhanced genomic benchmark, which consolidates 31 datasets across 11 representative tasks, with input sequence lengths ranging from 50 to 6,000 bp. Experimental results demonstrate that PlantBiMoE achieves the best performance on 20 out of 31 datasets and the average best when comparing with existing models. In summary, all above results demonstrate that our model can effectively represent plant genomic sequences, serving as a robust computational tool for diverse genomic tasks, while making substantive contributions to plant genomics, gene editing, and synthetic biology. The code is available at: https://github.com/HUST-Keep-Lin/PlantBiMoE