Benchmarking LLM-based agents for single-cell omics analysis

TL;DR

This paper introduces a comprehensive benchmarking system for evaluating LLM-based agents in single-cell omics analysis, highlighting their capabilities, limitations, and the importance of code quality and self-reflection.

Contribution

It presents a novel, unified benchmarking platform with multidimensional metrics and diverse tasks, establishing a standard for assessing AI agents in computational biology.

Findings

Grok3-beta achieves state-of-the-art performance.

Multi-agent frameworks improve collaboration and efficiency.

Code quality and self-reflection are key to success.

Abstract

Background: The surge in single-cell omics data exposes limitations in traditional, manually defined analysis workflows. AI agents offer a paradigm shift, enabling adaptive planning, executable code generation, traceable decisions, and real-time knowledge fusion. However, the lack of a comprehensive benchmark critically hinders progress. Results: We introduce a novel benchmarking evaluation system to rigorously assess agent capabilities in single-cell omics analysis. This system comprises: a unified platform compatible with diverse agent frameworks and LLMs; multidimensional metrics assessing cognitive program synthesis, collaboration, execution efficiency, bioinformatics knowledge integration, and task completion quality; and 50 diverse real-world single-cell omics analysis tasks spanning multi-omics, species, and sequencing technologies. Our evaluation reveals that Grok3-beta achieves state-of-the-art performance among tested agent frameworks. Multi-agent frameworks significantly enhance collaboration and execution efficiency over single-agent approaches through specialized role division. Attribution analyses of agent capabilities identify that high-quality code generation is crucial for task success, and self-reflection has the most significant overall impact, followed by retrieval-augmented generation (RAG) and planning. Conclusions: This work highlights persistent challenges in code generation, long-context handling, and context-aware knowledge retrieval, providing a critical empirical foundation and best practices for developing robust AI agents in computational biology.