Beyond the Singular: Revealing the Value of Multiple Generations in Benchmark Evaluation

TL;DR

This paper introduces a hierarchical statistical model that leverages multiple generations from large language models to improve benchmark evaluation accuracy, reduce variance, and provide detailed prompt-level insights.

Contribution

It presents a novel statistical framework that accounts for LLM randomness and enhances benchmark reliability through multiple generations and detailed prompt analysis.

Findings

Using multiple generations improves benchmark score estimation accuracy.

The model enables prompt-level difficulty scoring based on correct ratios.

A data map visualizes prompt difficulty and semantics for error detection.

Abstract

Large language models (LLMs) have demonstrated significant utility in real-world applications, exhibiting impressive capabilities in natural language processing and understanding. Benchmark evaluations are crucial for assessing the capabilities of LLMs as they can provide a comprehensive assessment of their strengths and weaknesses. However, current evaluation methods often overlook the inherent randomness of LLMs by employing deterministic generation strategies or relying on a single random sample, resulting in unaccounted sampling variance and unreliable benchmark score estimates. In this paper, we propose a hierarchical statistical model that provides a more comprehensive representation of the benchmarking process by incorporating both benchmark characteristics and LLM randomness. We show that leveraging multiple generations improves the accuracy of estimating the benchmark score and reduces variance. Multiple generations also allow us to define $\mathbb P\left(\text{correct}\right)$, a prompt-level difficulty score based on correct ratios, providing fine-grained insights into individual prompts. Additionally, we create a data map that visualizes difficulty and semantics of prompts, enabling error detection and quality control in benchmark construction.