On the Calibration of Large Language Models and Alignment

TL;DR

This paper systematically examines how training processes and settings affect the calibration of large language models across generation, factuality, and understanding, addressing reliability concerns in widespread LLM applications.

Contribution

It provides a comprehensive analysis of calibration throughout the entire training process of aligned language models, highlighting factors influencing their reliability.

Findings

Calibration varies with training stages and settings

Model size and data influence calibration quality

Insights into calibration for generation, factuality, and understanding

Abstract

As large language models attract increasing attention and find widespread application, concurrent challenges of reliability also arise at the same time. Confidence calibration, an effective analysis method for gauging the reliability of deep models, serves as a crucial tool for assessing and improving their reliability. However, such investigation has been comparatively underexplored. In this work, we conduct a systematic examination of the calibration of aligned language models throughout the entire construction process, including pretraining and alignment training. At each stage, we investigate how different training settings, such as parameter scales and training data, affect model calibration. To thoroughly assess model calibration, we evaluate models on three most concerned aspects: generation, factuality and understanding. Our work sheds light on whether popular LLMs are well-calibrated and how the training process influences model calibration.