Auto-Bench: An Automated Benchmark for Scientific Discovery in LLMs

TL;DR

Auto-Bench is a new standardized benchmark designed to evaluate Large Language Models' ability to perform scientific discovery tasks involving causal reasoning, hypothesis generation, and decision-making in natural and social sciences.

Contribution

The paper introduces Auto-Bench, the first comprehensive benchmark for assessing LLMs' scientific discovery capabilities through causal graph discovery tasks.

Findings

State-of-the-art LLMs perform poorly as problem complexity increases.

Significant performance gap exists between current LLMs and human scientists.

Auto-Bench enables systematic evaluation of LLMs' scientific reasoning abilities.

Abstract

Given the remarkable performance of Large Language Models (LLMs), an important question arises: Can LLMs conduct human-like scientific research and discover new knowledge, and act as an AI scientist? Scientific discovery is an iterative process that demands efficient knowledge updating and encoding. It involves understanding the environment, identifying new hypotheses, and reasoning about actions; however, no standardized benchmark specifically designed for scientific discovery exists for LLM agents. In response to these limitations, we introduce a novel benchmark, \textit{Auto-Bench}, that encompasses necessary aspects to evaluate LLMs for scientific discovery in both natural and social sciences. Our benchmark is based on the principles of causal graph discovery. It challenges models to uncover hidden structures and make optimal decisions, which includes generating valid justifications. By engaging interactively with an oracle, the models iteratively refine their understanding of underlying interactions, the chemistry and social interactions, through strategic interventions. We evaluate state-of-the-art LLMs, including GPT-4, Gemini, Qwen, Claude, and Llama, and observe a significant performance drop as the problem complexity increases, which suggests an important gap between machine and human intelligence that future development of LLMs need to take into consideration.