Annotation-Efficient Universal Honesty Alignment

TL;DR

This paper introduces EliCal, a two-stage, annotation-efficient framework for training large language models to recognize their knowledge limits and express calibrated confidence, using minimal supervision.

Contribution

EliCal combines inexpensive self-consistency supervision with a small set of correctness annotations to achieve universal honesty alignment in LLMs.

Findings

EliCal achieves near-optimal honesty alignment with only 1k annotations.

EliCal outperforms calibration-only baselines on unseen tasks.

HonestyBench provides a large-scale benchmark for honesty alignment.

Abstract

Honesty alignment-the ability of large language models (LLMs) to recognize their knowledge boundaries and express calibrated confidence-is essential for trustworthy deployment. Existing methods either rely on training-free confidence estimation (e.g., token probabilities, self-consistency) or training-based calibration with correctness annotations. While effective, achieving universal honesty alignment with training-based calibration requires costly, large-scale labeling. To support annotation-efficient training, we introduce Elicitation-Then-Calibration (EliCal), a two-stage framework that first elicits internal confidence using inexpensive self-consistency supervision, then calibrates this confidence with a small set of correctness annotations. To support a large-scale study, we release HonestyBench, a benchmark covering ten free-form QA datasets with 560k training and 70k evaluation instances annotated with correctness and self-consistency signals. Experiments show that EliCal achieves near-optimal alignment with only 1k correctness annotations (0.18% of full supervision) and better alignment performance on unseen MMLU tasks than the calibration-only baseline, offering a scalable solution toward universal honesty alignment in LLMs.