Eval Factsheets: A Structured Framework for Documenting AI Evaluations

TL;DR

Eval Factsheets provide a structured documentation framework for AI evaluations, enhancing transparency, reproducibility, and comparability across diverse evaluation methods and benchmarks.

Contribution

This paper introduces Eval Factsheets, a comprehensive taxonomy and questionnaire-based framework for systematically documenting AI evaluation methodologies.

Findings

Effectively captures diverse evaluation paradigms

Maintains consistency and comparability across evaluations

Enhances transparency and reproducibility in AI assessments

Abstract

The rapid proliferation of benchmarks has created significant challenges in reproducibility, transparency, and informed decision-making. However, unlike datasets and models -- which benefit from structured documentation frameworks like Datasheets and Model Cards -- evaluation methodologies lack systematic documentation standards. We introduce Eval Factsheets, a structured, descriptive framework for documenting AI system evaluations through a comprehensive taxonomy and questionnaire-based approach. Our framework organizes evaluation characteristics across five fundamental dimensions: Context (Who made the evaluation and when?), Scope (What does it evaluate?), Structure (With what the evaluation is built?), Method (How does it work?) and Alignment (In what ways is it reliable/valid/robust?). We implement this taxonomy as a practical questionnaire spanning five sections with mandatory and recommended documentation elements. Through case studies on multiple benchmarks, we demonstrate that Eval Factsheets effectively captures diverse evaluation paradigms -- from traditional benchmarks to LLM-as-judge methodologies -- while maintaining consistency and comparability. We hope Eval Factsheets are incorporated into both existing and newly released evaluation frameworks and lead to more transparency and reproducibility.