LABBench2: An Improved Benchmark for AI Systems Performing Biology Research

TL;DR

LABBench2 is an advanced benchmark with nearly 1,900 tasks designed to evaluate AI systems' real-world scientific research capabilities, showing significant difficulty and room for improvement over previous benchmarks.

Contribution

It introduces LABBench2, an improved, more challenging benchmark for assessing AI's practical scientific research abilities, expanding on prior work with more realistic tasks.

Findings

Current models show substantial performance gaps on LABBench2.

LABBench2's difficulty range indicates significant room for AI development.

Performance improvements over LAB-Bench are notable but still incomplete.

Abstract

Optimism for accelerating scientific discovery with AI continues to grow. Current applications of AI in scientific research range from training dedicated foundation models on scientific data to agentic autonomous hypothesis generation systems to AI-driven autonomous labs. The need to measure progress of AI systems in scientific domains correspondingly must not only accelerate, but increasingly shift focus to more real-world capabilities. Beyond rote knowledge and even just reasoning to actually measuring the ability to perform meaningful work. Prior work introduced the Language Agent Biology Benchmark LAB-Bench as an initial attempt at measuring these abilities. Here we introduce an evolution of that benchmark, LABBench2, for measuring real-world capabilities of AI systems performing useful scientific tasks. LABBench2 comprises nearly 1,900 tasks and is, for the most part, a continuation of LAB-Bench, measuring similar capabilities but in more realistic contexts. We evaluate performance of current frontier models, and show that while abilities measured by LAB-Bench and LABBench2 have improved substantially, LABBench2 provides a meaningful jump in difficulty (model-specific accuracy differences range from -26% to -46% across subtasks) and underscores continued room for performance improvement. LABBench2 continues the legacy of LAB-Bench as a de facto benchmark for AI scientific research capabilities and we hope that it continues to help advance development of AI tools for these core research functions. To facilitate community use and development, we provide the task dataset at https://huggingface.co/datasets/futurehouse/labbench2 and a public eval harness at https://github.com/EdisonScientific/labbench2.