On Meta-Evaluation

TL;DR

This paper introduces a formal framework and benchmark for meta-evaluation, systematically comparing evaluation methods across domains, revealing limitations and proposing a unified sampling approach that improves reliability.

Contribution

It provides the first large-scale quantitative comparison of evaluation methods and introduces AxiaBench, a benchmark for meta-evaluation across multiple domains.

Findings

No single evaluation method achieves both accuracy and efficiency across all scenarios.

Existing methods like DoE and observational designs often deviate from ground truth.

A unified stratified sampling approach outperforms prior methods in all tested domains.

Abstract

Evaluation is the foundation of empirical science, yet the evaluation of evaluation itself -- so-called meta-evaluation -- remains strikingly underdeveloped. While methods such as observational studies, design of experiments (DoE), and randomized controlled trials (RCTs) have shaped modern scientific practice, there has been little systematic inquiry into their comparative validity and utility across domains. Here we introduce a formal framework for meta-evaluation by defining the evaluation space, its structured representation, and a benchmark we call AxiaBench. AxiaBench enables the first large-scale, quantitative comparison of ten widely used evaluation methods across eight representative application domains. Our analysis reveals a fundamental limitation: no existing method simultaneously achieves accuracy and efficiency across diverse scenarios, with DoE and observational designs in particular showing significant deviations from real-world ground truth. We further evaluate a unified method of entire-space stratified sampling from previous evaluatology research, and the results report that it consistently outperforms prior approaches across all tested domains. These results establish meta-evaluation as a scientific object in its own right and provide both a conceptual foundation and a pragmatic tool set for advancing trustworthy evaluation in computational and experimental research.