From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipeline

TL;DR

This paper presents BenchBuilder, an automated pipeline leveraging LLMs to curate high-quality, challenging benchmarks from crowd-sourced datasets, enabling scalable, continuous evaluation of large language models with high alignment to human preferences.

Contribution

The introduction of BenchBuilder, an automated, LLM-based pipeline for scalable benchmark curation and evaluation, reducing manual effort and costs.

Findings

BenchBuilder successfully curated Arena-Hard-Auto with 500 challenging prompts.

The benchmark achieves 3x higher separation of model performances compared to MT-Bench.

It correlates 98.6% with human preference rankings at a cost of $20.

Abstract

The rapid evolution of Large Language Models (LLMs) has outpaced the development of model evaluation, highlighting the need for continuous curation of new, challenging benchmarks. However, manual curation of high-quality, human-aligned benchmarks is expensive and time-consuming. To address this, we introduce BenchBuilder, an automated pipeline that leverages LLMs to curate high-quality, open-ended prompts from large, crowd-sourced datasets, enabling continuous benchmark updates without human in the loop. We apply BenchBuilder to datasets such as Chatbot Arena and WildChat-1M, extracting challenging prompts and utilizing LLM-as-a-Judge for automatic model evaluation. To validate benchmark quality, we propose new metrics to measure a benchmark's alignment with human preferences and ability to separate models. We release Arena-Hard-Auto, a benchmark consisting 500 challenging prompts curated by BenchBuilder. Arena-Hard-Auto provides 3x higher separation of model performances compared to MT-Bench and achieves 98.6% correlation with human preference rankings, all at a cost of $20. Our work sets a new framework for the scalable curation of automated benchmarks from extensive data.