The limits of bio-molecular modeling with large language models : a cross-scale evaluation

TL;DR

This paper systematically evaluates large language models across multiple bio-molecular tasks, revealing their strengths and limitations in understanding complex biological systems.

Contribution

It introduces BioMol-LLM-Bench, a comprehensive benchmark with 26 tasks across four difficulty levels for evaluating LLMs in bio-molecular modeling.

Findings

Chain-of-thought data offers limited or negative benefits.

Hybrid mamba-attention architectures excel in long sequences.

Supervised fine-tuning enhances specialization but reduces generalization.

Abstract

The modeling of bio-molecular system across molecular scales remains a central challenge in scientific research. Large language models (LLMs) are increasingly applied to bio-molecular discovery, yet systematic evaluation across multi-scale biological problems and rigorous assessment of their tool-augmented capabilities remain limited. We reveal a systematic gap between LLM performance and mechanistic understanding through the proposed cross-scale bio-molecular benchmark: BioMol-LLM-Bench, a unified framework comprising 26 downstream tasks that covers 4 distinct difficulty levels, and computational tools are integrated for a more comprehensive evaluation. Evaluation on 13 representative models reveals 4 main findings: chain-of-thought data provides limited benefit and may even reduce performance on biological tasks; hybrid mamba-attention architectures are more effective for long bio-molecular sequences; supervised fine-tuning improves specialization at the cost of generalization; and current LLMs perform well on classification tasks but remain weak on challenging regression tasks. Together, these findings provide practical guidance for future LLM-based modeling of molecular systems.