LMR-BENCH: Evaluating LLM Agent's Ability on Reproducing Language Modeling Research

TL;DR

This paper introduces LMR-BENCH, a comprehensive benchmark to evaluate large language models' ability to accurately reproduce code from scientific NLP research papers, revealing significant current limitations.

Contribution

The paper presents LMR-BENCH, the first benchmark specifically designed to assess LLMs' performance in reproducing research code from scientific publications.

Findings

State-of-the-art LLMs show limited accuracy in code reproduction

Models struggle with complex scientific reasoning tasks

Significant gaps remain in LLMs' ability to autonomously reproduce research code

Abstract

Large language model (LLM) agents have demonstrated remarkable potential in advancing scientific discovery. However, their capability in the fundamental yet crucial task of reproducing code from research papers, especially in the NLP domain, remains underexplored. This task includes unique complex reasoning challenges in the intellectual synthesis of abstract concepts and the comprehension of code repositories with interdependent files. Motivated by this gap, we present LMR-BENCH, a benchmark designed to systematically evaluate the capability of LLM agents on code reproduction from Language Modeling Research. It consists of 28 code reproduction tasks derived from 23 research papers published in top-tier NLP venues over the past five years, spanning nine fundamental categories. Models are provided with a research paper, a code repository containing one or more masked functions, and instructions for implementing these functions. We conduct extensive experiments in standard prompting and LLM agent settings with state-of-the-art LLMs, evaluating the accuracy of unit tests and performing LLM-based evaluation of code correctness. Experimental results reveal that even the most advanced models still exhibit persistent limitations in scientific reasoning and code synthesis, highlighting critical gaps in LLM agents' ability to autonomously reproduce scientific research