CodonMoE: DNA Language Models for mRNA Analyses

TL;DR

CodonMoE is a lightweight adapter that transforms DNA language models into effective RNA analyzers, achieving state-of-the-art results across multiple tasks with fewer parameters and maintaining computational efficiency.

Contribution

We introduce CodonMoE, a universal adapter that enables DNA language models to analyze RNA without RNA-specific pretraining, reducing complexity and improving performance.

Findings

Outperforms unmodified DNA models on RNA prediction tasks

Achieves state-of-the-art results with 80% fewer parameters

Maintains sub-quadratic complexity while enhancing performance

Abstract

Genomic language models (gLMs) face a fundamental efficiency challenge: either maintain separate specialized models for each biological modality (DNA and RNA) or develop large multi-modal architectures. Both approaches impose significant computational burdens - modality-specific models require redundant infrastructure despite inherent biological connections, while multi-modal architectures demand massive parameter counts and extensive cross-modality pretraining. To address this limitation, we introduce CodonMoE (Adaptive Mixture of Codon Reformative Experts), a lightweight adapter that transforms DNA language models into effective RNA analyzers without RNA-specific pretraining. Our theoretical analysis establishes CodonMoE as a universal approximator at the codon level, capable of mapping arbitrary functions from codon sequences to RNA properties given sufficient expert capacity. Across four RNA prediction tasks spanning stability, expression, and regulation, DNA models augmented with CodonMoE significantly outperform their unmodified counterparts, with HyenaDNA+CodonMoE series achieving state-of-the-art results using 80% fewer parameters than specialized RNA models. By maintaining sub-quadratic complexity while achieving superior performance, our approach provides a principled path toward unifying genomic language modeling, leveraging more abundant DNA data and reducing computational overhead while preserving modality-specific performance advantages.