Beyond Surface Statistics: Robust Conformal Prediction for LLMs via Internal Representations

TL;DR

This paper introduces a conformal prediction framework for large language models that leverages internal layer-wise representations to improve uncertainty estimation, especially under distribution shifts.

Contribution

It proposes Layer-Wise Information scores as nonconformity measures, enhancing validity and efficiency in LLM question answering across various benchmarks.

Findings

Improved validity-efficiency trade-off over surface-level methods.

Better reliability under cross-domain shifts.

Maintains competitive in-domain calibration.

Abstract

Large language models are increasingly deployed in settings where reliability matters, yet output-level uncertainty signals such as token probabilities, entropy, and self-consistency can become brittle under calibration--deployment mismatch. Conformal prediction provides finite-sample validity under exchangeability, but its practical usefulness depends on the quality of the nonconformity score. We propose a conformal framework for LLM question answering that uses internal representations rather than output-facing statistics: specifically, we introduce Layer-Wise Information (LI) scores, which measure how conditioning on the input reshapes predictive entropy across model depth, and use them as nonconformity scores within a standard split conformal pipeline. Across closed-ended and open-domain QA benchmarks, with the clearest gains under cross-domain shift, our method achieves a better validity--efficiency trade-off than strong text-level baselines while maintaining competitive in-domain reliability at the same nominal risk level. These results suggest that internal representations can provide more informative conformal scores when surface-level uncertainty is unstable under distribution shift.