Efficient Benchmarking of Language Models

TL;DR

This paper addresses the high computational costs of language model benchmarking by proposing methods to reduce evaluation costs while maintaining reliability, demonstrated through the HELM benchmark with significant cost savings.

Contribution

It introduces the concept of Efficient Benchmarking, proposes a new measure DIoR to assess reliability impacts, and presents an algorithm for cost-effective evaluation of language models.

Findings

Benchmark ranking can change by removing low-ranked models.

Evaluating only a fraction of examples can preserve ranking accuracy.

Proposed algorithm reduces computation by over 100 times with minimal reliability loss.

Abstract

The increasing versatility of language models (LMs) has given rise to a new class of benchmarks that comprehensively assess a broad range of capabilities. Such benchmarks are associated with massive computational costs, extending to thousands of GPU hours per model. However, the efficiency aspect of these evaluation efforts had raised little discussion in the literature. In this work, we present the problem of Efficient Benchmarking, namely, intelligently reducing the computation costs of LM evaluation without compromising reliability. Using the HELM benchmark as a test case, we investigate how different benchmark design choices affect the computation-reliability trade-off. We propose to evaluate the reliability of such decisions, by using a new measure -- Decision Impact on Reliability, DIoR for short. We find, for example, that a benchmark leader may change by merely removing a low-ranked model from the benchmark, and observe that a correct benchmark ranking can be obtained by considering only a fraction of the evaluation examples. Based on our findings, we outline a set of concrete recommendations for efficient benchmark design and utilization practices. To take a step further, we use our findings to propose an evaluation algorithm, that, when applied to the HELM benchmark, leads to dramatic cost savings with minimal loss of benchmark reliability, often reducing computation by x100 or more.