# Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning

## TL;DR

This paper evaluates 21 RNA language models to understand their strengths and weaknesses in capturing RNA structure, function, and evolution without fine-tuning.

## Contribution

A standardized zero-shot benchmark for RNA language models across structural, functional, and evolutionary tasks.

## Key findings

- RNA-specific pretraining is essential for capturing structural information.
- Evolutionary signals from multiple sequence alignments improve model performance.
- Model scaling alone does not guarantee better performance; architecture and objectives matter.

## Abstract

RNA language models (LMs) are increasingly applied to RNA structure and function analysis, yet their intrinsic representational capacities remain poorly characterized. Here, we present a standardized zero-shot evaluation of 21 RNA LMs, with representative DNA LMs included as reference controls. Three complementary tasks—attention-based RNA secondary structure prediction, embedding-based RNA classification, and mutational fitness estimation from sequence likelihoods—are evaluated without downstream fine-tuning. Our results reveal substantial variability across models and clear trade-offs between structural, functional, and evolutionary representations. RNA-specific, noncoding RNA-enriched pretraining is crucial for capturing structural information, while evolutionary signals from multiple sequence alignments substantially boost performance. Although model scaling yields gains, architectural and objective choices critically influence performance across task categories. Together, this study provides a foundational benchmark, highlights inherent challenges in learning unified RNA representations, and offers insights for developing next-generation RNA foundation models.