GRACE: A Granular Benchmark for Evaluating Model Calibration against Human Calibration

TL;DR

GRACE is a benchmark that evaluates language model calibration by comparing model responses to human responses across gradually revealing clues, enabling granular analysis of calibration errors.

Contribution

We introduce GRACE, a novel benchmark with a new metric CalScore, for detailed evaluation of language model calibration against human behavior.

Findings

Humans are better calibrated than models despite lower accuracy.

State-of-the-art models struggle on GRACE, indicating calibration challenges.

GRACE effectively measures progress in model calibration improvements.

Abstract

Language models are often miscalibrated, leading to confidently incorrect answers. We introduce GRACE, a benchmark for language model calibration that incorporates comparison with human calibration. GRACE consists of question-answer pairs, in which each question contains a series of clues that gradually become easier, all leading to the same answer; models must answer correctly as early as possible as the clues are revealed. This setting permits granular measurement of model calibration based on how early, accurately, and confidently a model answers. After collecting these questions, we host live human vs. model competitions to gather 1,749 data points on human and model teams' timing, accuracy, and confidence. We propose a metric, CalScore, that uses GRACE to analyze model calibration errors and identify types of model miscalibration that differ from human behavior. We find that although humans are less accurate than models, humans are generally better calibrated. Since state-of-the-art models struggle on GRACE, it effectively evaluates progress on improving model calibration.