Mitigating LLM Hallucinations via Conformal Abstention

TL;DR

This paper introduces a conformal abstention method for large language models that reliably reduces hallucinations by determining when the model should abstain, using self-evaluation and conformal prediction for theoretical guarantees.

Contribution

It proposes a novel abstention procedure leveraging self-evaluation and conformal prediction to control hallucination rates with theoretical guarantees.

Findings

Effectively bounds hallucination rates on question answering datasets

Maintains lower abstention rates on long response datasets

Achieves comparable accuracy on short answer datasets

Abstract

We develop a principled procedure for determining when a large language model (LLM) should abstain from responding (e.g., by saying "I don't know") in a general domain, instead of resorting to possibly "hallucinating" a non-sensical or incorrect answer. Building on earlier approaches that use self-consistency as a more reliable measure of model confidence, we propose using the LLM itself to self-evaluate the similarity between each of its sampled responses for a given query. We then further leverage conformal prediction techniques to develop an abstention procedure that benefits from rigorous theoretical guarantees on the hallucination rate (error rate). Experimentally, our resulting conformal abstention method reliably bounds the hallucination rate on various closed-book, open-domain generative question answering datasets, while also maintaining a significantly less conservative abstention rate on a dataset with long responses (Temporal Sequences) compared to baselines using log-probability scores to quantify uncertainty, while achieveing comparable performance on a dataset with short answers (TriviaQA). To evaluate the experiments automatically, one needs to determine if two responses are equivalent given a question. Following standard practice, we use a thresholded similarity function to determine if two responses match, but also provide a method for calibrating the threshold based on conformal prediction, with theoretical guarantees on the accuracy of the match prediction, which might be of independent interest.