BenchmarkCards: Standardized Documentation for Large Language Model Benchmarks

TL;DR

BenchmarkCards provides a standardized documentation framework for LLM benchmarks, improving transparency, comparability, and ease of selection for users by systematically capturing key benchmark attributes.

Contribution

It introduces a validated, standardized documentation framework for LLM benchmarks, addressing complexity and transparency issues in benchmark selection.

Findings

Simplifies benchmark selection process.

Enhances transparency and comparability.

Validated through user studies.

Abstract

Large language models (LLMs) are powerful tools capable of handling diverse tasks. Comparing and selecting appropriate LLMs for specific tasks requires systematic evaluation methods, as models exhibit varying capabilities across different domains. However, finding suitable benchmarks is difficult given the many available options. This complexity not only increases the risk of benchmark misuse and misinterpretation but also demands substantial effort from LLM users, seeking the most suitable benchmarks for their specific needs. To address these issues, we introduce \texttt{BenchmarkCards}, an intuitive and validated documentation framework that standardizes critical benchmark attributes such as objectives, methodologies, data sources, and limitations. Through user studies involving benchmark creators and users, we show that \texttt{BenchmarkCards} can simplify benchmark selection and enhance transparency, facilitating informed decision-making in evaluating LLMs. Data & Code: https://github.com/SokolAnn/BenchmarkCards