AbGen: Evaluating Large Language Models in Ablation Study Design and Evaluation for Scientific Research

TL;DR

AbGen is a benchmark for evaluating large language models' ability to design ablation studies in scientific research, revealing current models' limitations and the unreliability of automated evaluation methods.

Contribution

This paper introduces AbGen, the first benchmark for assessing LLMs in designing scientific ablation studies, and develops AbGen-Eval to evaluate automated assessment reliability.

Findings

LLMs lag behind human experts in ablation study design quality.

Current automated evaluation methods are unreliable for complex scientific tasks.

Developed AbGen-Eval to measure the reliability of LLM performance assessments.

Abstract

We introduce AbGen, the first benchmark designed to evaluate the capabilities of LLMs in designing ablation studies for scientific research. AbGen consists of 1,500 expert-annotated examples derived from 807 NLP papers. In this benchmark, LLMs are tasked with generating detailed ablation study designs for a specified module or process based on the given research context. Our evaluation of leading LLMs, such as DeepSeek-R1-0528 and o4-mini, highlights a significant performance gap between these models and human experts in terms of the importance, faithfulness, and soundness of the ablation study designs. Moreover, we demonstrate that current automated evaluation methods are not reliable for our task, as they show a significant discrepancy when compared to human assessment. To better investigate this, we develop AbGen-Eval, a meta-evaluation benchmark designed to assess the reliability of commonly used automated evaluation systems in measuring LLM performance on our task. We investigate various LLM-as-Judge systems on AbGen-Eval, providing insights for future research on developing more effective and reliable LLM-based evaluation systems for complex scientific tasks.