Spark-LLM-Eval: A Distributed Framework for Statistically Rigorous Large Language Model Evaluation

TL;DR

Spark-LLM-Eval is a scalable, distributed framework built on Apache Spark that enables statistically rigorous evaluation of large language models across massive datasets, with efficient caching and comprehensive statistical analysis.

Contribution

It introduces a distributed evaluation system that handles large-scale datasets, incorporates statistical rigor, and optimizes cost through response caching, all implemented as open-source software.

Findings

Achieves linear scaling with cluster size in benchmarks.

Provides bootstrap confidence intervals and significance tests for metrics.

Enables cost-effective evaluation via content-addressable caching.

Abstract

Evaluating large language models at scale remains a practical bottleneck for many organizations. While existing evaluation frameworks work well for thousands of examples, they struggle when datasets grow to hundreds of thousands or millions of samples. This scale is common when assessing model behavior across diverse domains or conducting comprehensive regression testing. We present Spark-LLM-Eval, a distributed evaluation framework built natively on Apache Spark. The system treats evaluation as a data-parallel problem, partitioningexamplesacrossexecutorsandaggregatingresultswithproperstatistical accounting. Beyond raw throughput, we emphasize statistical rigor: every reported metric includes bootstrap confidence intervals, and model comparisons come with appropriate significance tests (paired t-tests, McNemar's test, or Wilcoxon signed-rank, depending on the metric type). The framework also addresses the cost problem inherent in LLM evaluation through content-addressable response caching backed by Delta Lake, which allows iterating on metric definitions without re-running inference. We describe the system architecture, the statistical methodology, and report benchmark results showing linear scaling with cluster size. The framework and all evaluation code are available as open source.