EvalCards: A Framework for Standardized Evaluation Reporting

TL;DR

This paper introduces EvalCards, a standardized framework aimed at improving transparency, reproducibility, and governance in NLP evaluation reporting through a practical disclosure tool.

Contribution

The paper proposes EvalCards, a new framework for evaluation reporting that addresses current shortcomings in reproducibility, accessibility, and governance in NLP.

Findings

EvalCards enhance transparency for researchers and practitioners.

They provide a practical foundation for governance compliance.

The framework addresses key shortcomings in current reporting practices.

Abstract

Evaluation has long been a central concern in NLP, and transparent reporting practices are more critical than ever in today's landscape of rapidly released open-access models. Drawing on a survey of recent work on evaluation and documentation, we identify three persistent shortcomings in current reporting practices: reproducibility, accessibility, and governance. We argue that existing standardization efforts remain insufficient and introduce Evaluation Disclosure Cards (EvalCards) as a path forward. EvalCards are designed to enhance transparency for both researchers and practitioners while providing a practical foundation to meet emerging governance requirements.