# TECP: Token-Entropy Conformal Prediction for LLMs

## TL;DR

TECP introduces a token-entropy based conformal prediction framework for large language models, providing reliable uncertainty quantification with formal coverage guarantees without requiring internal model details.

## Contribution

It proposes a novel token-entropy measure for uncertainty and integrates it into a conformal prediction pipeline, enabling black-box LLM uncertainty quantification with provable guarantees.

## Key findings

- Achieves reliable coverage across six LLMs and two benchmarks.
- Produces compact prediction sets outperforming prior methods.
- Provides a principled, efficient approach for trustworthy black-box LLM generation.

## Abstract

Uncertainty quantification (UQ) for open-ended language generation remains a critical yet underexplored challenge, especially under black-box constraints where internal model signals are inaccessible. In this paper, we introduce Token-Entropy Conformal Prediction (TECP), a novel framework that leverages token-level entropy as a logit-free, reference-free uncertainty measure and integrates it into a split conformal prediction (CP) pipeline to construct prediction sets with formal coverage guarantees. Unlike existing approaches that rely on semantic consistency heuristics or white-box features, TECP directly estimates epistemic uncertainty from the token entropy structure of sampled generations and calibrates uncertainty thresholds via CP quantiles to ensure provable error control. Empirical evaluations across six large language models and two benchmarks (CoQA and TriviaQA) demonstrate that TECP consistently achieves reliable coverage and compact prediction sets, outperforming prior self-consistency-based UQ methods. Our method provides a principled and efficient solution for trustworthy generation in black-box LLM settings.