Evaluating and Calibrating LLM Confidence on Questions with Multiple Correct Answers

TL;DR

This paper investigates the challenge of confidence calibration in large language models for questions with multiple correct answers, introduces a new benchmark called MACE, and proposes a novel aggregation method, SCA, to improve calibration accuracy.

Contribution

The paper reveals the failure of existing calibration methods with multiple answers, introduces MACE benchmark, and proposes SCA for better confidence estimation in such scenarios.

Findings

Calibration worsens as answer count increases.

SCA outperforms existing methods in mixed-answer calibration.

Accuracy improves with more answers, but confidence estimates decline.

Abstract

Confidence calibration is essential for making large language models (LLMs) reliable, yet existing training-free methods have been primarily studied under single-answer question answering. In this paper, we show that these methods break down in the presence of multiple valid answers, where disagreement among equally correct responses leads to systematic underestimation of confidence. To enable a systematic study of this phenomenon, we introduce MACE, a benchmark of 12,000 factual questions spanning six domains with varying numbers of correct answers. Experiments across 15 representative calibration methods and four LLM families (7B-72B) reveal that while accuracy increases with answer cardinality, estimated confidence consistently decreases, causing severe miscalibration for questions with mixed answer counts. To address this issue, we propose Semantic Confidence Aggregation (SCA), which aggregates confidence over multiple high-probability sampled responses. SCA achieves state-of-the-art calibration performance under mixed-answer settings while preserving strong calibration on single-answer questions.