Benchmarking virtual cell models for in-the-wild perturbation response

TL;DR

This paper introduces a standardized benchmarking framework for virtual cell models, evaluating their robustness and generalization in realistic, complex biological scenarios to improve their practical utility.

Contribution

It presents a modular evaluation setup that assesses virtual cell models under challenging, real-world conditions, revealing their limitations and guiding future improvements.

Findings

Model performance varies significantly across different biological contexts.

Performance drops under strict evaluation conditions, indicating limited robustness.

Different metrics lead to different model rankings, affecting interpretation.

Abstract

Virtual cell (VC) models aim to predict cellular responses to any perturbations in silico and have emerged as a promising approach for drug discovery and precision medicine. Yet, a clear gap still remains: while models routinely reported impressive results on standard benchmarks, it is unclear whether their predictions are truly meaningful in practice. This is mainly due to limitations in current evaluation setups, which are often overly simplified or inconsistent, and do not reflect the complexity and variability of real biological systems. Here, we introduce a standardized and modular benchmarking framework for virtual cell prediction. Our framework evaluates diverse models under in-the-wild challenging scenarios, including unseen cell contexts, unseen perturbations, and cross-dataset generalization, which better reflect practical applications. Our analysis shows that model performance is highly context-dependent and shaped by task design and evaluation criteria. In commonly used setups, performance is often overestimated, and naive dataset aggregation can even reduce performance. When evaluated under more strict conditions, model performance drops markedly, indicating limited robustness to shifts across cellular contexts. In unseen perturbation settings, models including simple linear approaches capture global transcriptional trends but fail to recover fine-grained perturbation-specific effects. In addition, different evaluation metrics focus on different biological properties, leading to substantially different model rankings. Together, our framework provides a more reliable and biologically grounded evaluation, offering clearer guidance for applying virtual cell models in real scenarios.