Trust in One Round: Confidence Estimation for Large Language Models via Structural Signals

TL;DR

This paper introduces Structural Confidence, a novel, single-pass, model-agnostic method that improves LLM output correctness estimation by analyzing internal structural signals, outperforming traditional confidence estimators across diverse tasks.

Contribution

The work presents a new framework leveraging multi-scale structural signals from LLMs' hidden states, enabling efficient, robust confidence estimation without multiple stochastic samples or auxiliary models.

Findings

Outperforms baselines in AUROC and AUPR across four benchmarks

Uses a single deterministic pass for confidence estimation

Effective across diverse, domain-specific tasks

Abstract

Large language models (LLMs) are increasingly deployed in domains where errors carry high social, scientific, or safety costs. Yet standard confidence estimators, such as token likelihood, semantic similarity and multi-sample consistency, remain brittle under distribution shift, domain-specialised text, and compute limits. In this work, we present Structural Confidence, a single-pass, model-agnostic framework that enhances output correctness prediction based on multi-scale structural signals derived from a model's final-layer hidden-state trajectory. By combining spectral, local-variation, and global shape descriptors, our method captures internal stability patterns that are missed by probabilities and sentence embeddings. We conduct extensive, cross-domain evaluation across four heterogeneous benchmarks-FEVER (fact verification), SciFact (scientific claims), WikiBio-hallucination (biographical consistency), and TruthfulQA (truthfulness-oriented QA). Our Structural Confidence framework demonstrates strong performance compared with established baselines in terms of AUROC and AUPR. More importantly, unlike sampling-based consistency methods which require multiple stochastic generations and an auxiliary model, our approach uses a single deterministic forward pass, offering a practical basis for efficient, robust post-hoc confidence estimation in socially impactful, resource-constrained LLM applications.