Report Cards: Qualitative Evaluation of Language Models Using Natural Language Summaries

TL;DR

This paper introduces 'report cards', human-readable summaries of language model capabilities, offering a more interpretable and holistic evaluation method that complements traditional benchmarks.

Contribution

It proposes a novel framework for generating and evaluating report cards based on specificity, faithfulness, and interpretability, with an iterative algorithm for automatic creation.

Findings

Report cards provide insights beyond traditional benchmarks.

The iterative algorithm effectively generates accurate report cards.

Evaluation criteria ensure clarity and relevance for human understanding.

Abstract

The rapid development and dynamic nature of large language models (LLMs) make it difficult for conventional quantitative benchmarks to accurately assess their capabilities. We propose report cards, which are human-interpretable, natural language summaries of model behavior for specific skills or topics. We develop a framework to evaluate report cards based on three criteria: specificity (ability to distinguish between models), faithfulness (accurate representation of model capabilities), and interpretability (clarity and relevance to humans). We also propose an iterative algorithm for generating report cards without human supervision and explore its efficacy by ablating various design choices. Through experimentation with popular LLMs, we demonstrate that report cards provide insights beyond traditional benchmarks and can help address the need for a more interpretable and holistic evaluation of LLMs.