Towards Comprehensive Benchmarking Infrastructure for LLMs In Software Engineering

TL;DR

This paper introduces BEHELM, a comprehensive benchmarking infrastructure designed to evaluate large language models in software engineering across multiple tasks and quality dimensions, addressing current evaluation gaps.

Contribution

The paper presents BEHELM, a holistic, standardized benchmarking framework that unifies software-scenario specification with multi-metric evaluation for LLMs in software engineering.

Findings

Identified core barriers: lack of rich datasets, overreliance on ML metrics, and non-standardized data pipelines.

BEHELM enables multi-task, multi-metric assessment across diverse software engineering scenarios.

Aims to facilitate fair, realistic, and scalable evaluation of LLMs in software engineering.

Abstract

Large language models for code are advancing fast, yet our ability to evaluate them lags behind. Current benchmarks focus on narrow tasks and single metrics, which hide critical gaps in robustness, interpretability, fairness, efficiency, and real-world usability. They also suffer from inconsistent data engineering practices, limited software engineering context, and widespread contamination issues. To understand these problems and chart a path forward, we combined an in-depth survey of existing benchmarks with insights gathered from a dedicated community workshop. We identified three core barriers to reliable evaluation: the absence of software-engineering-rich datasets, overreliance on ML-centric metrics, and the lack of standardized, reproducible data pipelines. Building on these findings, we introduce BEHELM, a holistic benchmarking infrastructure that unifies software-scenario specification with multi-metric evaluation. BEHELM provides a structured way to assess models across tasks, languages, input and output granularities, and key quality dimensions. Our goal is to reduce the overhead currently required to construct benchmarks while enabling a fair, realistic, and future-proof assessment of LLMs in software engineering.