Delta -- Contrastive Decoding Mitigates Text Hallucinations in Large Language Models

TL;DR

Delta is an inference-time technique that reduces hallucinations in large language models by contrasting output distributions with masked inputs, improving factual accuracy without retraining.

Contribution

The paper introduces Delta, a novel inference-only method that mitigates hallucinations in LLMs by contrasting outputs for original and masked prompts, requiring no retraining or extra data.

Findings

Improves SQuAD v1.1 and v2 accuracy by 3-6 percentage points.

Enhances TriviaQA and Natural Questions scores by 2-7 points.

Increases SQuAD v2 no-answer exact match by over 10 percentage points.

Abstract

Large language models (LLMs) demonstrate strong capabilities in natural language processing but remain prone to hallucinations, generating factually incorrect or fabricated content. This issue undermines their reliability, particularly in high-stakes domains such as healthcare and legal advisory. To address this challenge, we propose Delta, an inference-time method that reduces hallucinations without requiring model retraining or additional data. Delta works by randomly masking parts of the input prompt and contrasting the output distributions for the original and masked inputs, effectively suppressing hallucinations through inference-only computations. We evaluate Delta on context-rich question-answering benchmarks, achieving absolute improvements of approximately 3 and 6 percentage points on SQuAD v1.1 and v2, respectively, and 7 and 2 percentage points on TriviaQA and Natural Questions under-sampling decoding. Delta also improves the no-answer exact match score on SQuAD v2 by over ten percentage points, demonstrating its effectiveness in mitigating hallucinations arising from contextual ambiguity. These results highlight Delta as a computationally efficient and scalable approach for improving the reliability of LLMs in real-world applications.