Chance-Constrained Inference for Hallucination Risk Control in Large Language Models

TL;DR

This paper introduces a method called chance-constrained inference that explicitly controls the probability of hallucinations in large language models during deployment, ensuring more reliable and safe outputs.

Contribution

It formulates hallucination risk as a probabilistic constraint and develops an adaptive inference procedure to enforce these constraints efficiently.

Findings

Reliable hallucination risk control demonstrated on NaturalQuestions-based tasks.

Early detection of infeasible inputs improves safety.

Method outperforms confidence-based baselines in providing probabilistic guarantees.

Abstract

Large language models generate outputs stochastically and may produce fluent but invalid responses, including factual hallucinations. Existing mitigation strategies reduce average error rates but do not provide explicit control over the \emph{frequency} of such failures under repeated use. We formulate inference as a deployment-time risk control problem and introduce \emph{chance-constrained inference}, which directly bounds the probability of hallucinations among accepted generations. Hallucinations are modeled as stochastic constraint violations, and we show that confidence-based selective prediction does not, in general, imply probabilistic risk guarantees. To enforce chance constraints efficiently, we propose a sequential, anytime-valid inference procedure that adaptively certifies feasibility or infeasibility using finite samples, avoiding conservative fixed-sample bounds. Experiments on questions inspired by NaturalQuestions and controlled multi-hop question answering demonstrate reliable risk control, early detection of intrinsically infeasible inputs, and safe composition under repeated use, while confidence-based baselines fail to provide consistent guarantees.