Decoding Memories: An Efficient Pipeline for Self-Consistency Hallucination Detection

TL;DR

This paper introduces a Decoding Memory Pipeline (DMP) that enhances the efficiency of self-consistency hallucination detection in large language models by reducing redundant computations, achieving up to 3x speedup.

Contribution

It presents a novel approach to identify and eliminate redundancy in self-consistency methods, significantly improving generation speed without losing accuracy.

Findings

Achieves up to 3x speedup in generation efficiency

Maintains AUROC performance comparable to baseline methods

Applicable to various tasks beyond hallucination detection

Abstract

Large language models (LLMs) have demonstrated impressive performance in both research and real-world applications, but they still struggle with hallucination. Existing hallucination detection methods often perform poorly on sentence-level generation or rely heavily on domain-specific knowledge. While self-consistency approaches help address these limitations, they incur high computational costs due to repeated generation. In this paper, we conduct the first study on identifying redundancy in self-consistency methods, manifested as shared prefix tokens across generations, and observe that non-exact-answer tokens contribute minimally to the semantic content. Based on these insights, we propose a novel Decoding Memory Pipeline (DMP) that accelerates generation through selective inference and annealed decoding. Being orthogonal to the model, dataset, decoding strategy, and self-consistency baseline, our DMP consistently improves the efficiency of multi-response generation and holds promise for extension to alignment and reasoning tasks. Extensive experiments show that our method achieves up to a 3x speedup without sacrificing AUROC performance.

Tables

Table 1. Table 1: AUROC, mean AUROC, running time, and reuse ratio of different baselines using LLaMA-2-7B-Chat and Mistral-7B-Instruct on TriviaQA, NQ Open, SQuAD, and HaluEval. The best mean AUROC is highlighted in bold.

	Llama2-7B-Chat				Mistral-7B-Instruct
	TriviaQA	NQ Open	SQuAD	HaluEval	TriviaQA	NQ Open	SQuAD	HaluEval
LN-Entopy	69.5	61.2	66.9	68.2	73.2	65.7	59.9	71.2
LN-Entropy+DMP	71.0	64.4	68.6	69.8	73.3	67.8	61.0	73.8
Lexical Similarity	66.7	62.9	70.0	61.5	75.8	68.2	64.8	79.3
Lexical Similarity+DMP	67.7	62.0	70.8	65.3	75.5	67.8	63.9	78.0
Semantic Entropy	86.4	81.8	82.5	77.9	84.6	79.9	76.7	84.3
Semantic Entropy+DMP	85.4	82.0	81.5	77.2	84.8	78.9	77.1	84.9
Semantic Entropy-B	86.3	81.6	82.2	78.0	84.8	80.0	77.0	84.5
Semantic Entropy-B+DMP	86.4	82.3	81.4	77.0	84.6	79.1	77.5	84.8
SelfCheckGPT	64.9	64.9	70.5	61.2	71.4	64.5	65.1	74.6
SelfCheckGPT+DMP	67.5	62.9	70.3	64.5	71.6	66.9	60.1	76.0
EigenScore	70.6	65.8	70.0	66.3	75.1	66.5	65.2	79.9
EigenScore+DMP	69.5	63.9	71.8	68.3	75.8	67.3	66.4	78.4
EigenScore-B	74.2	68.5	72.7	75.0	79.0	73.4	63.7	84.0
EigenScore-B+DMP	71.3	65.8	73.7	75.1	78.9	71.4	65.1	81.8
Mean – Baseline	74.09	69.53	73.54	69.73	77.70	71.17	67.49	79.69
Mean – DMP (Ours)	74.11	69.04	74.01	71.03	77.79	71.31	67.30	79.68
Time (s) – Baseline	1778	2368	1932	2148	1124	1750	1430	1225
Time (s) – DMP (Ours)	627 (2.8x)	1246 (1.9x)	1022 (1.9x)	904 (2.4x)	442 (2.5x)	927 (1.9x)	744 (1.9x)	630 (2.0x)
Reuse Ratio	66.8%	52.7%	50.4%	62.6%	63.4%	51.3%	51.3%	55.1%

Table 2. Table 2: AUROC and reuse ratio for technique components.

	Baseline	+Selective Inference	+Hard Decoding	+Annealed Decoding
AUROC	70.1	70.2	67.7	69.7
Reuse Ratio	–	56.9%	63.4%	66.8%

Table 3. Table 3: AUROC, mean AUROC, running time, and reuse ratio of different baselines using LLaMA-2-13B-Chat and Falcon-7B-Instruct on TriviaQA, NQ Open, SQuAD, and HaluEval. The best mean AUROC is highlighted in bold.

	Llama2-13B-Chat				Falcon-7B-Instruct
	TriviaQA	NQ Open	SQuAD	HaluEval	TriviaQA	NQ Open	SQuAD	HaluEval
LN-Entopy	0.686	0.594	0.697	0.666	0.608	0.603	0.507	0.659
LN-Entropy+DMP	0.713	0.636	0.736	0.686	0.621	0.638	0.573	0.689
Lexical Similarity	0.652	0.588	0.730	0.574	0.689	0.628	0.627	0.767
Lexical Similarity+DMP	0.646	0.599	0.727	0.599	0.692	0.618	0.623	0.746
SelfCheckGPT	0.633	0.623	0.753	0.605	0.651	0.597	0.623	0.732
SelfCheckGPT+DMP	0.641	0.632	0.742	0.624	0.672	0.616	0.642	0.693
EigenScore	0.701	0.625	0.776	0.657	0.717	0.642	0.703	0.786
EigenScore+DMP	0.702	0.628	0.754	0.660	0.710	0.630	0.647	0.768
EigenScore-B	0.725	0.647	0.754	0.701	0.725	0.666	0.728	0.799
EigenScore-B+DMP	0.718	0.641	0.737	0.698	0.724	0.664	0.690	0.776
Mean – Baseline	0.679	0.615	0.742	0.641	0.678	0.627	0.637	0.749
Mean – DMP (Ours)	0.684	0.627	0.739	0.653	0.684	0.633	0.635	0.734
Reuse Ratio	0.679	0.515	0.499	0.636	0.639	0.521	0.523	0.594

Table 4. Table 4: AUROC, mean AUROC, running time, and reuse ratio of different baselines using LLaMA-2-7B-Chat and Mistral-7B-Instruct on SciQ. The best mean AUROC is highlighted in bold.

	Llama2- 7B-Chat	Mistral- 7B-Instruct
LN-Entopy	62.6	62.6
LN-Entropy+DMP	66.5	60.3
Lexical Similarity	63.3	63.3
Lexical Similarity+DMP	65.1	68.6
EigenScore	68.4	68.4
EigenScore+DMP	67.2	69.8
EigenScore-B	67.7	67.7
EigenScore-B+DMP	66.6	72.1
Mean – Baseline	65.5	65.5
Mean – DMP (Ours)	66.35	67.7
Reuse Ratio	65.5%	57.1%

Equations

$${\mathbf{s}}_i=f_{\theta }(\mathbf{x},{\mathbf{y}}_{1:i-1}),i=1,2,\dots ,L,$$

$$\mathbf{P}({\mathbf{y}}_i\mid \mathbf{x},{\mathbf{y}}_{1:i-1})=\text{softmax}\left(\frac{{\mathbf{s}}_i}{T}\right),i=1,2,\dots ,L,$$

$${\mathbf{s}}_i=f_{\theta }(\mathbf{x},y_{i-1},{\mathbf{K}}_{1:i-2},{\mathbf{V}}_{1:i-2}),i=2,3,\dots ,L.$$

$$S_{\text{LNE}}(\mathcal{Y}|\mathbf{x})=-{\mathbb{E}}_{\mathbf{y}\in \mathcal{Y}}\frac{1}{L_{\mathbf{y}}}\sum _{i=1}^{L_{\mathbf{y}}}\log \mathbf{P}({\mathbf{y}}_i\mid \mathbf{x},{\mathbf{y}}_{1:i-1}),$$

$${\mathbf{M}}_m=({\mathbf{y}}^{(m)},{\mathbf{s}}^{(m)}),$$

$${\mathbf{M}}_m=({\mathbf{y}}^{(m)},{\mathbf{s}}^{(m)},{\mathbf{K}}^{(m)},{\mathbf{V}}^{(m)}).$$

$${\mu }_r(B_G)$$

$${\mu }_r({\mu }_a)$$

$$I_i^{(m)}=-\cos \left(\mathbf{h}({\mathbf{x}}_{-1}),\mathbf{h}({\mathbf{y}}_i^{(m)})\right),$$

$${\mathcal{I}}^{(m)}=\{i\mid I_i^{(m)}<\frac{\alpha }{L^{(m)}}\sum _{i=1}^{L^{(m)}}{I}_i^{(m)}\},$$

$${\mathbf{s}}_i^{(m)}=\eta \ast {\mathbf{s}}_i^{(m)},\text{if }i\in {\mathcal{I}}^{(m)},$$

$${\mathbf{s}}_i^{(n)}={\mathbf{y}}_i^{(m)},$$

$$\text{and }{\mathbf{y}}_i^{(m)}=\arg \max {\mathbf{s}}_i^{(m)},$$

$${\text{Attn}}_i=\text{softmax}\left(\frac{{\mathbf{Q}}_i{\mathbf{K}}_{1:i}^T}{\sqrt{d_k}}\right){\mathbf{V}}_{1:i},$$

$${\mathbf{Q}}_i={\mathbf{X}}_i{\mathbf{W}}^Q,{\mathbf{K}}_{1:i}={\mathbf{X}}_{1:i}{\mathbf{W}}^K,{\mathbf{V}}_{1:i}={\mathbf{X}}_{1:i}{\mathbf{W}}^V,$$

$${\mathbf{s}}_i=f_{\theta }(\mathbf{x},y_{i-1},{\mathbf{K}}_{1:i-2},{\mathbf{V}}_{1:i-2}),i=2,3,\dots ,L.$$

$$S_{\text{LNE}}(\mathcal{Y}|\mathbf{x})=-{\mathbb{E}}_{\mathbf{y}\in \mathcal{Y}}\frac{1}{L_{\mathbf{y}}}\sum _{i=1}^{L_{\mathbf{y}}}\log \mathbf{P}({\mathbf{y}}_i\mid \mathbf{x},{\mathbf{y}}_{1:i-1}),$$

$$S_{\text{LEX}}(\mathcal{Y}|\mathbf{x})=\frac{1}{N(N-1)}\sum _{i=1}^N\sum _{j=i+1}^N\text{sim}({\mathbf{y}}_i,{\mathbf{y}}_j),$$

$$S_{\text{BERT}}(\mathcal{Y}|\mathbf{x})=1-\frac{1}{N}\sum _{i=1}^N\mathcal{B}({\mathbf{y}}_i,\hat{\mathbf{y}}),$$

$$\mathbf{Z}=\text{Concat}([{\mathbf{h}}_1[-1],{\mathbf{h}}_2[-1],\dots ,{\mathbf{h}}_N[-1]]),$$

$$\Sigma ={\mathbf{Z}}^{\top }(\mathbf{I}-\frac{1}{N}{1}^{\top }1)\mathbf{Z},$$

$$S_{\text{Eigen}}=\frac{1}{N}\sum _{i=1}^N\log \det (\Sigma +\alpha \mathbf{I}),$$

$$S_{\text{SE}}(\mathcal{Y}|\mathbf{x})=-\sum _{c\in \mathcal{C}}\mathbf{P}(c|\mathbf{x})\log \mathbf{P}(c|\mathbf{x}),$$

$$P(c|\mathbf{x})=\sum _{{\mathbf{y}}_i\in c}\mathbf{P}({\mathbf{y}}_i|\mathbf{x})$$

$$p_{\text{sentence}}=$$

$$\ast \sum _{m=1}^M\sum _{i=1}^N\sum _{j=i+1}^{N}1\{\text{Entail}({\mathbf{y}}_i^{(m)},{\mathbf{y}}_j^{(m)})\},$$

$$p_{\text{word}}=$$

$$\ast \sum _{m=1}^M\sum _{i=1}^N\sum _{j=i+1}^{N}1\{\text{Entail}({\mathbf{y}}_i^{(m)},{\mathbf{y}}_j^{(m)})\}$$

$$\ast \frac{\min (L_{{\mathbf{y}}_i^{(m)}},L_{{\mathbf{y}}_j^{(m)}}}{{\sum }_{m=1}^M{\sum }_{i=1}^N{L}_{{\mathbf{y}}_i^{(m)}}},$$

Full text

11footnotetext: Preprint. Under review. This version: August 28, 2025.

Code — https://anonymous.4open.science/status/Decoding-Memorie-Pipeline-4E8F

1 Introduction

Large Language Models (LLMs) have made significant advancements in both research and practical applications (OpenAI 2023; AI@Meta 2024). However, a persistent challenge is their tendency to hallucinate, where the model produces outputs that are grammatically correct and fluent, yet factually inaccurate, especially in sentence-level generation (Park et al. 2025; Ji et al. 2023). The difficulty of identifying factual error poses a substantial barrier to deploying LLMs in reliability-critical domains such as scientific research and medical diagnostics (Athaluri et al. 2023; Pal, Umapathi, and Sankarasubbu 2023; Chai et al. 2024). Therefore, it is urgent to design methods that accurately detect hallucination and reject unreliable answers (Manakul, Liusie, and Gales 2023; Chen et al. 2024).

Recent research has explored various approaches for detecting hallucinations in LLM outputs. One line of work estimates the likelihood of generated content using metrics such as perplexity and energy scores (Ren et al. 2022; Liu et al. 2020). However, the scores using likelihood are affected by perturbations in grammar and common word usage, making them less effective at accurately identifying hallucinations at the sentence level (Chen et al. 2024). Another approach involves linear probing of hidden states (Azaria and Mitchell 2023; Li et al. 2025), nevertheless, requiring labeled training data and often struggling to generalize to out-of-distribution examples (Pan et al. 2023; Manakul, Liusie, and Gales 2023). Moreover, due to the overconfident nature of LLMs, these methods fail to provide reliable confidence estimates for hallucination detection (Xiong et al. 2023).

Given the limitations of existing approaches, self-consistency methods have garnered significant attention for hallucination detection (Farquhar et al. 2024; Chen et al. 2024). These methods assess the consistency among multiple responses to the same query, using metrics such as semantic entropy. They not only achieve strong performance in detecting hallucinations but also exhibit robustness across diverse domains (Manakul, Liusie, and Gales 2023). Moreover, the ensemble nature enables uncertainty quantification, allowing them to produce confidence scores for hallucination detection. However, a key drawback of self-consistency methods is their high computational cost, as they require generating multiple responses per query. It often incurs 5 to 10 times the cost, making them less suitable for resource-constrained applications (Farquhar et al. 2024).

In this work, we present the first preliminary study that investigates the redundancy inherent in self-consistency methods based on multiple generations. We observe that, for a broad range of questions, the generated multiple responses often share similar tokens and prefixes, indicating overlap in token sequences. Furthermore, non-exact-answer tokens that do not contribute substantively to the core content of the answer, such as common function words “a” and “the”, have minimal impact on consistency evaluation. Therefore, the redundancy is therein though responses share different template caused by those tokens. We identify that two kinds of redundancy are widely present across models and datasets, constituting a substantial portion of the generated content.

To exploit the redundancy in multiple generations, we propose a Decoding Memory Pipeline (DMP) to accelerate self-consistency methods. Specifically, we introduce selective inference to leverage the autoregressive pattern of language models, which skips redundant forward passes by reusing cached computations from overlapped prefixes across generations. To further increase the portion of computation that can be skipped, we propose annealed decoding, which progressively lowers the sampling temperature to make non-exact-answer tokens more deterministic during generation. Our DMP is agnostic to self-consistency methods, enabling broad applicability for hallucination detection. To evaluate the effectiveness of our proposed DMP, we conduct extensive experiments on several benchmark datasets, including TriviaQA, NQ-Open, SQuAD, and HaluEval using a range of strong baselines. Our results show that DMP adequately reduces the redundancy, and accelerate the generation process by up to 3x without compromising hallucination detection performance. In summary, our main contributions are as follows:

• •

To the best of our knowledge, we are the first to identify redundancy in self-consistency methods in hallucination detection. Specifically, we reveal that redundancy arises from shared prefix tokens and non-exact-answer tokens across models and datasets.

• •

We propose a novel Decoding Memory Pipeline (DMP) that is agnostic to self-consistency methods, enabling accelerated generation. Specifically, we introduce selective inference, which skips redundant computations by leveraging repeated tokens, and annealed decoding, which increases token reuse by mitigating the impact of non-exact-answer tokens during generation.

• •

Extensive experiments on TriviaQA, NQ-Open, SQuAD, and HaluEval with multiple self-consistency methods demonstrate that our DMP accelerates generation by up to 3x without compromising hallucination detection performance across multiple self-consistency baselines.

2 Related Works

2.1 Efficient LLMs

LLMs face significant challenges in efficient deployment, prompting extensive research into improving inference efficiency (Wan et al. 2023). To address the quadratic complexity of the attention mechanism, key-value (KV) cache has become a standard inference strategy, enabling reuse of past key-value pairs (Hooper et al. 2024; Liu et al. 2024). Additionally, attention masks are employed to facilitate batch-wise inference (Vaswani et al. 2017; Devlin et al. 2019). Various approaches, such as pruning (Ma, Fang, and Wang 2023), quantization (Gao et al. 2025; Lin et al. 2024), and knowledge distillation (Xu et al. 2024), have been proposed to further accelerate inference. However, these methods are generally not tailored to self-consistency techniques, which incur computational redundancy due to generating multiple responses for the same input prompt. Our DMP is compatible with both KV caching and batch-wise inference, and is orthogonal to existing inference acceleration techniques.

2.2 Hallucination Detection

Hallucinations in LLMs present major challenges for reliable deployment, leading to growing interest in effective detection methods (Ji et al. 2023; Rawte, Sheth, and Das 2023). One common approach estimates hallucination scores by computing generation likelihoods, using metrics such as perplexity or energy scores (Ren et al. 2022; Liu et al. 2020). However, these methods often fail to achieve satisfactory detection performance. Other works attempt to identify hallucinations by training classifiers on hidden states (Azaria and Mitchell 2023; Li et al. 2025; Su et al. 2024; Orgad et al. 2024), but these approaches require labeled data and struggle with out-of-domain generalization (Pan et al. 2023; Manakul, Liusie, and Gales 2023). Furthermore, the overconfidence of LLMs limits the reliability of such confidence-based detection (Xue et al. 2023). Self-consistency methods, such as lexical similarities, SelfCheckGPT, and semantic entropy, offer an alternative by evaluating consistency across multiple responses to the same query (Manakul, Liusie, and Gales 2023; Farquhar et al. 2024; Chen et al. 2024; Lin, Liu, and Shang 2022). Nevertheless, these methods are hindered by their high computational cost.

3 Background

In this section, we provide essential background on LLM generation and self-consistency methods.

3.1 LLM Generation Pipeline

LLMs typically adopt an autoregressive generation paradigm, which involves alternating between inference and decoding steps (Henighan et al. 2020). In the inference stage, the model computes a logit for the next token, conditioned on the input context and all previously tokens. Given the input prompt ${\mathbf{x}}$, the logit of generating the $i$-th token ${\mathbf{y}}_{i}$ is computed as follows:

[TABLE]

where $f_{\theta}$ denotes a transformer-based model parameterized by $\theta$, and $L$ is the total number of tokens to be generated. In the decoding stage, a multinomial distribution over the token ${\mathbf{y}}_{i}$ is constructed based on its logit. Given a sampling temperature $T$, the probability is computed as:

[TABLE]

where the next token ${\mathbf{y}}_{i}$ is sampled from this distribution. The process repeats until an end-of-sequence token is produced. To trade off between diversity and determinism, various sampling strategies are employed, including greedy decoding, beam search, top-$k$, and top-$p$ sampling.

KV Cache. To reduce the computational burden of the attention mechanism, past key ${\mathbf{K}}_{1:i-1}$ and value tensors ${\mathbf{V}}_{1:i-1}$ are cached to avoid redundant computation. The forward process with KV cache is updated as:

[TABLE]

Despite being optimized by KV cache, the sequential nature of generation remains computationally intensive, as it requires $O(L)$ model forward passes for a sequence of length $L$. For more details, please refer to Appendix A.

3.2 Self-Consistency Hallucination Detection

Despite their impressive performance across a wide range of applications, LLMs are prone to hallucinations, where the responses appear plausible but contain factual inaccuracies. To address this issue, self-consistency methods have been proposed, which not only detect hallucination without training but also provide uncertainty of the prediction. Given an input prompt ${\mathbf{x}}$, suppose we generate $N$ responses to evaluate consistency, denoted as a set ${\mathcal{Y}}=\{{\mathbf{y}}_{1},{\mathbf{y}}_{2},\dots,{\mathbf{y}}_{N}\}$. A consistency score $S({\mathcal{Y}};{\mathbf{x}})$ is then computed to quantify the agreement among these responses. For example, the Length-Normalized Entropy evaluates the normalized likelihood of the generated responses (Malinin and Gales 2020):

[TABLE]

where a higher consistency score means a more factual answer. However, generating multiple responses further increases the computational cost of hallucination detection.

4 Decoding Memory Pipeline

In this section, we introduce our decoding memory pipeline. We begin by presenting preliminary studies that reveal the computational redundancy in self-consistency generation. To address this, we propose a selective inference mechanism that reduces computational cost by selectively skipping forward passes using cached logits in Section 4.2. We further introduce annealed decoding, a technique designed to enhance token reuse by progressively lowering the sampling temperature for non-exact-answer tokens in Section 4.3. An overview of our proposed pipeline is provided in Figure 1.

4.1 Redundancy in Self-Consistency Generation

We conduct preliminary studies to investigate computational redundancy in self-consistency hallucination detection. Specifically, we make the following two observations when generating multiple responses.

Observation 1: Repeated Prefix in Multiple Responses.

Intuitively, responses generated for the same input prompt often share overlapping content, which is dependent on the model confidence. Since we are working with autoregressive models and aim to exploit this redundancy, we focus on the repetition of prefixes, i.e., the initial segments of the responses. For example, given the question “What is the capital of Brazil?”, multiple responses commonly share the prefix “The capital of Brazil is” as shown in Figure 2(a).

To verify the presence of redundancy, we analyze prefix overlap in multiple-response generation using Llama2-7B-Chat (Touvron et al. 2023) across several datasets. We evaluate overlap from two perspectives: (1) the proportion of response pairs that share prefixes, and (2) the proportion of words in the shared prefixes relative to the total word count. As shown in Figure 2, the sentence-level and the word-level ratio exceeds $70\%$ and $25\%$ across all four datasets, respectively. These results demonstrate that redundancy is prevalent across both datasets and models. For additional implementation details and results on other models, please refer to Appendix B.1 and D.1.

Observation 2: Non-exact-answer tokens do not matter.

Recent research has shown that extracting exact answers is critical for improving performance in many tasks (Orgad et al. 2024; Asai and Hajishirzi 2020), as they capture the key information within responses. Consequently, hallucination detection should primarily rely on the exact answers, while non-exact-answer tokens affect sentence structure or word order. As shown in Figure 2(b), given the question What is the capital of Brazil?, the non-exact-answer tokens vary, but the responses convey the same semantic information, indicating the inner redundancy.

To test the generality of this assumption, we conduct experiments to assess the influence of non-exact-answer tokens. Specifically, for each question, we generate multiple responses and then group them based on their answer templates using GPT-3.5 (OpenAI 2023). The rationale is that non-exact-answer tokens primarily affect the template or word order of the responses. After clustering the responses, we compare the accuracy of the two groups. As shown in Figure 2(d), the accuracies of different groups are nearly identical to the overall accuracy, a trend consistent across datasets. This suggests that non-exact-answer tokens have a limited impact on the semantics of the generated responses. For implementation details, please refer to Appendix B.1.

4.2 Selective Inference

To mitigate the redundancy identified in Observation 1, we propose a selective inference strategy that skips unnecessary generation steps. The high-level motivation is that LLM inference will generate the same probability distribution with the same previous tokens, given the autoregressive pattern in Equation 1. To exploit this, we define a memory list ${\mathcal{M}}$ to store the response memory, which is initialized as an empty list. Then we define response memories to cache generated responses, where each entry stores the generated tokens ${\mathbf{y}}$ and their associated logits ${\mathbf{s}}$:

[TABLE]

where $m$ denotes the index of a cached generated response. During the generation of token ${\mathbf{y}}_{i}^{(n)}$, we iterate over the memory entries ${\mathbf{M}}_{m}\in{\mathcal{M}}$ to check whether the prefix ${\mathbf{y}}_{1:i-1}^{(n)}$ matches any stored response ${\mathbf{y}}_{1:i-1}^{(m)}$. If a match is found, we directly assign the corresponding ${\mathbf{s}}_{i}^{(m)}$ to ${\mathbf{y}}_{i}^{(n)}$, and reuse ${\mathbf{s}}_{i}^{(m)}$ to predict the next token, bypassing the model inference step. If no matching prefix is found, a forward pass is performed: ${\mathbf{s}}_{i}^{(n)}=f_{\theta}({\mathbf{x}},{\mathbf{y}}_{1:i-1}^{(n)})$. After generating a newly complete response ${\mathbf{y}}^{(n)}$, it will be added to the memory list ${\mathcal{M}}$ if it is not already present.

KV Cache. Selective inference remains compatible with KV caching by caching the key and value tensors of responses. Specifically, we augment the response memory with keys ${\mathbf{K}}^{(m)}$ and values ${\mathbf{V}}^{(m)}$ of the whole sequence:

[TABLE]

We assign ${\mathbf{K}}_{i}^{(m)}$ and ${\mathbf{V}}_{i}^{(m)}$ to ${\mathbf{y}}_{i}^{(n)}$ if a matching prefix is found in the memory set ${\mathcal{M}}$. If no match is found, a forward pass is performed using the KV cache: ${\mathbf{s}}_{i}^{(n)}=f_{\theta}({\mathbf{x}},{\mathbf{y}}_{i-1}^{(n)},{\mathbf{K}}_{1:i-2}^{(n)},{\mathbf{V}}_{1:i-2}^{(n)})$. Note that the outputs produced by our selective inference method are equivalent to those generated by the standard generation pipeline.

Batch Mode. Our DMP supports batch-wise inference, aligning with the parallel processing characteristics of deep learning. This is achieved through a reusing mask, where the model continues its forward pass until no prefix match can be identified under this mask. To be specific, the reusing mask $\mu_{r}$ is initialized as the attention mask in causal language models (Vaswani et al. 2017), where only the input prompt tokens are unmasked. Let the generated batch be $B_{G}$, and a reference batch $B_{R}$ of the same size be initialized with all entries set to None. At each generation step, we first perform a batch-wise prefix match between $B_{G}$ and $B_{R}$ under the attention mask $\mu_{a}$. For $j$-th instance in the batch, the reusing mask $\mu_{r}$ is updated as follows:

• •

For instance $B_{G}[j]$ that fails to match, set all the items of $\mu_{r}[j]$ as [math] and update $B_{R}[j]$ with $\textit{Next}({\mathcal{M}}_{B_{G}[j]})$.

• •

For instance $B_{G}[j]$ that matches, $\mu_{r}[j]$ is updated to follow $\mu_{a}[j]$, with the next token position unmasked.

Here, ${\mathcal{M}}_{B_{G}[j]}$ is the memory list of $B_{G}[j]$, and Next$({\mathcal{M}})$ fetches the next cached response in ${\mathcal{M}}$. For the instances with a successful match, both the attention and generated batches are updated as follows:

[TABLE]

where $\mathrm{Update}(\cdot)$ denotes the next token information update in selective inference. If no match is found after iterating through all the caches in ${\mathcal{M}}_{B_{G}}$, we perform a forward process over the batch $f(B_{G},\mu_{a})$. Each newly generated complete response $B_{G}[j]$ is added to the memory list ${\mathcal{M}}_{B_{G}[j]}$ if it is not already present. The complete batch-wise inference process is outlined in Appendix C, where all steps are implemented in a fully parallelization manner.

4.3 Annealed Decoding

As indicated in Observation 2, non-exact tokens have minimal influence on semantics and hallucination detection performance. To enhance template consistency in responses, we introduce annealed decoding to mitigate the impact of non-exact-answer tokens. To implement this, we first identify less important tokens using the cosine similarity. Then, during decoding, we progressively lower the sampling temperature for these tokens based on how frequently they appear, effectively promoting the reuse ratio across generations.

To identify less important tokens, we compute the cosine similarity between each response token embedding and the embedding of the input prompt. Intuitively, tokens that exhibit high similarity to the prompt embedding are likely to be semantically entailed or repeated from the prompt itself, and thus contribute less new information. In contrast, tokens with low similarity are more likely to introduce novel content not directly present in the prompt, making them informative and important. Therefore, we treat high-similarity tokens as non-exact-answer tokens and target them for temperature annealing. First, we augment the response memory ${\mathbf{M}}_{m}$ by caching the hidden states ${\mathbf{h}}^{(m)}=[{\mathbf{h}}({\mathbf{x}}),{\mathbf{h}}({\mathbf{y}}^{(m)})]$, where ${\mathbf{h}}(\cdot)$ denotes the final-layer hidden embeddings. The importance score of a response token ${\mathbf{y}}_{i}^{(k)}$ is computed as:

[TABLE]

where we represent the input prompt using the embedding of its last token, ${\mathbf{h}}({\mathbf{x}}_{-1})$. Then we select less important tokens by comparing them with the average score:

[TABLE]

where the hyperparameter $\alpha$ controls the selection threshold. Based on the occurrence frequency of non-exact-answer tokens, we reduce the sampling randomness of them, as more frequent tokens tend to form a preferable template for generation. In particular, when selective inference skips generation using the $m$-th cached response and the $i$-th token is identified as a non-exact-answer token, we scale up and update the cached logits ${\mathbf{s}}^{(m)}$ as follows:

[TABLE]

where $\eta\in(1,\infty)$ is the annealing speed for controlling the strength of annealing. Our technique is equivalent to lowering the sampling temperature and encourages deterministic reuse of non-exact-answer tokens while preserving the informative parts of the sequence.

Hard Decoding. For tokens with highly concentrated sampling distributions during generation, resampling often results in negligible variation and does not alter the semantic outcome. To increase the reuse efficiency in such cases, we introduce hard sampling, which deterministically reuses the token prediction when the model is confident. Specifically, if the maximum sampling probability exceeds a threshold $\gamma$ and the cached token is correspond to the maximum, we sample the next token as follows:

[TABLE]

where the hyperparameter $\gamma$ controls the confidence threshold for applying hard decoding.

Remark 1**.**

Although our primary focus in this paper is on hallucination detection, the applicability of DMP extends beyond this use case. Self-consistency methods are broadly employed in tasks such as uncertainty quantification (Kuhn, Gal, and Farquhar 2023), alignment (Amini et al. 2024), and reasoning (Zuo et al. 2025) in LLMs.

5 Experiments

In this section, we first describe our experimental settings. We then evaluate the performance of our proposed DMP across multiple datasets using self-consistency methods, demonstrating its ability to accelerate inference without compromising accuracy. Furthermore, we conduct comprehensive ablation studies to analyze the contributions of selective inference and annealed decoding.

5.1 Experiment Settings

Datasets. We evaluate the effectiveness of our proposed DMP on four generative question-answering (QA) datasets: TriviaQA (Joshi et al. 2017), NQ Open (Kwiatkowski et al. 2019), SQuAD (Rajpurkar et al. 2016), and the QA subset of HaluEval (Li et al. 2023). To further demonstrate the robustness of our method, we include additional results on SciQ (Welbl, Liu, and Gardner 2017) in Appendix E, showcasing its performance on domains requiring specialized knowledge. For each dataset, we use 400 test examples randomly sampled from the original larger dataset.

Models. Following prior work on hallucination detection, we conduct experiments using widely adopted open-source LLMs, including LLaMA-2-7B-Chat (Touvron et al. 2023) and Mistral-7B-Instruct (Jiang et al. 2023). To demonstrate the scalability and generalization of our DMP, we present additional experiments on LLaMA-2-13B-Chat and Falcon-7B-Instruct (Penedo et al. 2023), provided in Appendix D.2.

Evaluation. Following Farquhar et al. (2024), we use GPT-4o (Hurst et al. 2024) to label the correctness of generated responses, as standard metrics often fall short for sentence-level evaluation. We assess detection performance using the area under the receiver operating characteristic curve (AUROC). To quantify the generation cost, we report the average generation time per input prompt on a single H100 GPU. To factor out hardware and system effects, we also report the reuse ratio, defined as the percentage of generated tokens for which the forward process is skipped, thereby reflecting the theoretical speedup.

Baselines. We evaluate our proposed DMP against several widely-used self-consistency baselines, including LN-Entropy(Malinin and Gales 2020), Lexical Similarity(Lin, Liu, and Shang 2022), Semantic Entropy(Farquhar et al. 2024), SelfCheckGPT(Manakul, Liusie, and Gales 2023), and EigenScore (Chen et al. 2024). Moreover, we provide additional experiments on the black box version of Semantic Entropy and EigenScore as“Semantic Entropy-B” and “EigenScore-B”. For detailed descriptions of these baselines, please refer to Appendix A.

Implementation. We implement our selective inference using the KV Cache variant. We adopt the following default settings. We set the generation temperature to 0.8. For annealed decoding, we set the importance-selection threshold $\alpha$ to $0.9$ and the annealing speed $\eta$ to $1.4$. For hard decoding, we set the confidence threshold $\gamma$ to $0.8$. For additional implementation details, please refer to Appendix B.2. For notational simplicity, we denote our method built upon a baseline implementation as “+DMP”. For example, when applied to LN-Entropy, we denote it as “LN-Entropy+DMP”.

5.2 Main Results

Table 1 reports the AUROC, mean AUROC, running time, and reuse ratio across different models and datasets, with the best mean performance highlighted in bold. Based on these results, we draw the following conclusions.

Detection Performance. Our method consistently maintains a high AUROC across baselines, models, and datasets, in some cases even slightly exceeding baseline performance. Specifically, the AUROC reduction is at most $2.9\%$ across all trials, while the mean AUROC reduction over all self-consistency methods is limited to just $0.5\%$, both well within an acceptable range. Notably, the mean AUROC with our DMP surpasses the baseline in 5 out of 8 trials, indicating that the introduced sampling randomness does not compromise, and may even enhance the detection performance.

Efficiency Improvement. Applying DMP consistently improves the efficiency of self-consistency methods, as confirmed by both theoretical analysis and empirical results. In terms of reuse ratio, our approach skips up to $66.8\%$ of token generation in TriviaQA with Llama2-7B-Chat, achieving a $3\times$ acceleration. The measured running time closely matches the theoretical speedup on hardware. While the gains are smaller on more challenging datasets such as NQ Open and SQuAD, our method still consistently delivers at least a $2\times$ acceleration in both theory and $1.9\times$ practice.

5.3 Ablation Study

In this section, we present comprehensive ablation studies to examine the contributions of individual components of DMP, reporting mean measurements over the baselines. We first analyze the effect of the hard decoding threshold, followed by evaluating the effectiveness of annealed decoding. Furthermore, we assess the effect of sampling temperature on acceleration performance in Appendix D.3 and provide memory analysis in Appendix F. We denote w/ SI as the method that applies selective inference, and w/ Hard denotes the method applying hard decoding. We further denote Baseline as standard generation. The small error bar indicates the reuse ratio of the corresponding setting.

Effectiveness of Hard Decoding. We conduct experiments on TriviaQA and SQuAD using Llama2-7B-Chat, examining the trade-off between AUROC and reuse ratio under different confidence thresholds $\gamma\in\{0.6,0.7,0.8,0.9\}$. As shown in Figure 3, AUROC remains robust across different values of $\gamma$, while the reuse ratio increases as $\gamma$ decreases. We conservatively select $\gamma=0.8$ as the final threshold, as it provides a favorable balance. Notably, hard decoding does not necessarily reduce performance, suggesting that existing self-consistency methods may be underconfident.

Effectiveness of Annealed Decoding. We conduct experiments on the rescaling hyperparameter $\eta\in\{1.1,1.2,1.3,1.4\}$ in annealed decoding to study its impact on TriviaQA using Llama2-7B-Chat under two confidence thresholds $\gamma$. As shown in Figure 4, annealed decoding yields improvements over hard decoding and can even approach baseline performance on both efficiency and detection performance. We attribute this to the rescaling, which makes non-exact-answer tokens more deterministic, while hard decoding further enhances consistency and mitigates underconfidence in non-exact-answer tokens.

Additionally, we conduct experiments on the selection threshold $\alpha\in\{0.8,0.9,1.0,1.1,1.2\}$ in annealed decoding to study its impact on TriviaQA using Llama2-7B-Chat under two rescaling parameters. As shown in Figure 5, setting $\alpha=1.2$ consistently maintains a high AUROC while achieving a favorable trade-off. Finally, we present performance improvement applying our techniques in Table 2.

6 Conclusion

In this work, we present the Decoding Memory Pipeline (DMP), a novel approach to accelerate hallucination detection in large language models by leveraging redundancy in self-consistency decoding. By identifying and reusing repeated tokens across generations, DMP significantly reduces inference cost while maintaining strong detection performance. Our experiments demonstrate up to a 3x speedup without compromising AUROC scores, making DMP practical for real-world applications. A limitation of our approach is the increased memory cost resulting from caching more responses than standard generation. Developing efficient cache storage strategies is left for future work.

7 Acknowledgment

This research was supported by the Graduate Research at ORNL (GRO) Internship Program at Oak Ridge National Laboratory and used resources of the Oak Ridge Leadership Computing Facility (OLCF), which is a DOE Office of Science User Facility at the Oak Ridge National Laboratory supported by the U.S. Department of Energy under Contract No. DE-AC05-00OR22725.

Appendix A Background

In this section, we present the background knowledge about KV cache and the self-consistency baselines used in our paper.

KV cache. To reduce the computational burden of the attention mechanism, past key and value tensors are cached to avoid redundant computation. During generation, the past tokens involve the logit computation of the next token $y_{i+1}$ via the attention:

[TABLE]

where $d_{K}$ denotes the dimensionality of the keys, ${\mathbf{X}}_{1:i}$ denotes the input of attention, and ${\mathbf{W}}^{Q}$, ${\mathbf{W}}^{K}$, and ${\mathbf{W}}^{V}$ represent the linear projections for the query, key, and value, respectively. Due to the causal mask in the attention mechanism, ${\mathbf{K}}_{1:i-1}$ and ${\mathbf{V}}_{1:i-1}$ remain fixed during generation. Thus, only $K_{i}$ and $V_{i}$ need to be computed at each step, followed by concatenation with ${\mathbf{K}}_{1:i-1}$ and ${\mathbf{V}}_{1:i-1}$. Consequently, we can update Equation 1 in the main paper as follows:

[TABLE]

Despite the optimizations introduced by KV cache, the sequential nature of generation remains computationally intensive, as it requires $O(L)$ model forward passes for a sequence of length $L$.

LN-Entropy. Given $N$ responses ${\mathcal{Y}}=\{{\mathbf{y}}_{1},{\mathbf{y}}_{2},\dots,{\mathbf{y}}_{N}\}$ and the input prompt ${\mathbf{x}}$, length-normalized entropy computes the consistency score as follows:

[TABLE]

where $L_{\mathbf{y}}$ denotes the length of the response ${\mathbf{y}}$. LN-Entropy computes the consistency score as the average log-likelihood of the responses, normalized by response length.

Lexical Similarity. Given $N$ responses ${\mathcal{Y}}=\{{\mathbf{y}}_{1},{\mathbf{y}}_{2},\dots,$ ${\mathbf{y}}_{N}\}$ and the input prompt ${\mathbf{x}}$, lexical similarity estimates the consistency score by constructing semantic graphs and measuring similarities based on the uncertainty of a sequence-to-sequence model as follows:

[TABLE]

where sim$(\cdot,\cdot)$ is the similarity function using semantic rules. We compute lexical similarity using the ROUGE metric.111https://huggingface.co/spaces/evaluate-metric/rouge

SelfCheckGPT. Given $N$ responses ${\mathcal{Y}}=\{{\mathbf{y}}_{1},{\mathbf{y}}_{2},\dots,$ ${\mathbf{y}}_{N}\}$ and the input prompt ${\mathbf{x}}$, SelfCheckGPT evaluates the consistency between generations using methods such as BERTScore, designed QA, NLI tasks, or n-gram models. In our work, we adopt BERTScore, which is computed as follows:

[TABLE]

where {\mathcal{B}}$$(\cdot,\cdot) represents the BERTScore computation function, and $\hat{\mathbf{y}}$ is the most probable answer generated with greedy decoding. We choose NLI-RoBERTa-Large as the backbone of the score computation222https://huggingface.co/sentence-transformers/nli-roberta-large.

EigenScore. Given $N$ responses ${\mathcal{Y}}=\{{\mathbf{y}}_{1},{\mathbf{y}}_{2},\dots,$ ${\mathbf{y}}_{N}\}$ and the input prompt ${\mathbf{x}}$, EigenScore measures the consistency between answers with the eigenvalues of the sentence embedding matrix. To be specific, the sentence embedding matrix is the stack of the hidden states of the last token:

[TABLE]

where Concat$(\cdot)$ denotes concatenation along the first dimension, and ${\mathbf{h}}_{i}[-1]$ refers to the hidden state of the last token in response $i$. The covariance matrix of ${\mathbf{Z}}$ is then computed as:

[TABLE]

where $\mathbf{1}$ denotes an all-ones column vector. The EigenScore is then computed using the determinant of $\Sigma$:

[TABLE]

where $\alpha$ is a regularization term introduced to mitigate issues arising from singular values.

In black-box scenarios, where the hidden states of LLMs are inaccessible, we employ a proxy model to compute the hidden states. We denote this variant as EigenScore-B, using NLI-RoBERTa-Large2 as the proxy model.

Semantic Entropy. Given $N$ responses ${\mathcal{Y}}=\{{\mathbf{y}}_{1},{\mathbf{y}}_{2},\dots,$ ${\mathbf{y}}_{N}\}$ and the input prompt ${\mathbf{x}}$, semantic entropy first clusters the responses in to clusters ${\mathcal{C}}$ with the help of other language models. Then the consistency score is computed through a Rao-Blackwellized Monte Carlo integration over the semantic equivalence classes ${\mathcal{C}}$:

[TABLE]

where $P(c|{\mathbf{x}})$ is normalized over the sampled clusters. We use GPT-3.5-Turbo to generate cluster responses. For the specific prompt, please refer to Appendix B.2.

In black-box scenarios, where the sentence likelihood of LLMs is inaccessible for the Rao-Blackwellized formulation, we estimate semantic entropy by counting the samples in each cluster. We denote this variant as Semantic Entropy-B.

Appendix B Implementation Details

In this section, we present the implementation details of the implementation of our preliminary study and main experiments.

B.1 Preliminary Study

We show how the prefix sharing proportion is computed and how we construct the prompt for splitting the answers into two groups.

Prefix Sharing Proportion. To compute the prefix shared proportion of Observation 1, we use the sentence level and word level, respectively. Given $M$ input prompts ${\mathbf{x}}^{(i)}$, we generate $N$ responses ${\mathbf{y}}_{j}^{(i)}$, where $i=1,2,\dots,M$, and $j=1,2,\dots,N$. The sentence-level prefix sharing proportion is computed as follows:

[TABLE]

where $\mathbf{1}$ is the indicator function and Entail(${\mathbf{y}}_{i}^{(m)},{\mathbf{y}}_{j}^{(m)}$) represents either ${\mathbf{y}}_{i}^{(m)}$ starts with ${\mathbf{y}}_{j}^{(m)}$ or ${\mathbf{y}}_{j}^{(m)}$ starts with ${\mathbf{y}}_{i}^{(m)}$. The word-level prefix sharing proportion is computed as:

[TABLE]

where $L_{{\mathbf{y}}_{i}^{(m)}}$ represents the length of the response ${\mathbf{y}}_{i}^{(m)}$. For each model and dataset, we generate 10 responses for 400 questions using a sampling temperature of $0.8$.

Grouping Prompt For Observation 2, we group responses into two groups according to the answer template. We use GPT-3.5-Turbo for grouping with the following prompt:

“Given the question, please split the given responses into two groups according to the structure or template of the response. Please return two index list for the two groups.

*Question: *{Question}

*Response 1: *{Response 1}

*Response 2: *{Response 2}

⋮

*Response 10: *{Response 10}

Returned Lists: ”

In our experiments, we sample 100 questions for each dataset and generate 10 responses for each question. To label the correctness of the response, we use GPT-3.5-Turbo and use the same labeling method as the main experiments (Appendix B.2).

B.2 Main Experiments

In this section, we present the implementation details of our main experiments, including the prompt design for sentence-level response generation, the prompt design for response labeling, the short-answer cut technique in annealed decoding, and the selection of specific hyperparameters.

Prompt for Sentence Generation. Following the approach of semantic entropy (Farquhar et al. 2024), we use the following prompt to generate sentence-level responses:

Answer the following question in a single brief but complete sentence.

*Question: *{question}

Answer:

Prompt for Response Labeling. Following the approach of semantic entropy (Farquhar et al. 2024), we use the following prompt for response labeling when the dataset contains only a single reference answer:

*We are assessing the quality of answers to the following question: *{question}

*The expected answer is: *{reference answer}

*The proposed answer is: *{predicted answer}

Within the context of the question, does the proposed answer mean the same as the expected answer? Respond only with yes or no.

For datasets with multiple reference answers, we employ the following prompt with small modifications:

*We are assessing the quality of answers to the following question: *{question}

The following are expected answers to this question:

*The proposed answer is: *{predicted answer}

Within the context of the question, does the proposed answer mean the same as any of the expected answers? Respond only with yes or no.

Prompt for Entailment Estimating. In semantic entropy, we need to cluster the responses. We use the following prompt with GPT-3.5-Turbo:

*We are evaluating answers to the question *{question}

Here are two possible answers:

*Possible Answer 1: *{text1}

*Possible Answer 2: *{text2}

Does Possible Answer 1 semantically entail Possible Answer 2?

Respond with entailment, contradiction, or neutral.

Dealing with Short Answer in Annealed Decoding. Our proposed annealed decoding method searches for non-exact-answer tokens. However, in some cases, the generated responses are already exact answers under our generation prompt. To avoid unnecessary errors, we identify such short answers and exclude them from annealed decoding. In our experiments, we define short answers as responses containing fewer than 10 tokens.

SQuAD Experimental Settings. For SQuAD, we find that experiments with annealed decoding perform significantly worse than those using only selective inference with hard decoding, which also provides greater efficiency in generation. Therefore, we report hard decoding results for our experiments on SQuAD. We attribute this outcome to the inherent difficulty of the dataset.

Appendix C Batch-Wise Algorithm

In this section, we present the algorithm for batch-wise selective inference in Algorithm 2. In the algorithm, Update($\cdot$) denotes the token-level information update in selective inference, while Next(${\mathcal{M}}$) retrieves the next cached response from ${\mathcal{M}}$. Note that although all update rules during generation are applied in parallel with the mask, we present them in an instance-wise manner for improved readability.

Appendix D Additional Results

In this section, we present additional experiments, including preliminary studies, main experiments on other models, and an ablation study on sampling temperature selection. Unless otherwise noted, the experimental settings are consistent with those used in the main experiments.

D.1 Preliminary Study on Other Models

As shown in the main paper, sentence-level and word-level prefix sharing frequently occurs in Llama2-7B-Chat across datasets. Here, we provide statistical results for other models in Figures 6, 7, and 8. The results indicate that redundancy is prevalent across different models.

D.2 Main Experiments on Other Models

We report experiments on LLaMA2-13B-Chat and Falcon-7B-Instruct to demonstrate the scalability and generalization of our DMP. To reduce the cost of OpenAI API usage, we omit the semantic entropy results. As shown in Table 3, our method consistently improves generation efficiency while maintaining high AUROC across both scalable models and others.

D.3 Ablation Study of Sampling Temperature

Sampling temperature can influence the performance of hallucination detection. A higher temperature is often desirable for capturing uncertainty, while our method instead leverages redundancy, where a lower temperature increases the proportion of response reuse. To study this trade-off, we conduct experiments with $T\in\{0.5,0.7,0.8,1.0\}$ on both the baseline and selective inference (denoted as w/ SI). As shown in Figure 9 and 10, selective inference maintains generation quality while substantially improving efficiency. For a conservative balance, we adopt $T=0.8$ as the final sampling temperature.

Appendix E Results on SciQ

To verify that our method is not domain-dependent, which is a key advantage of self-consistency methods, we conduct experiments on SciQ (Welbl, Liu, and Gardner 2017) using LLaMA2-7B-Chat and Mistral-7B-Instruct. To reduce time and cost, we evaluate only four baselines: LN-Entropy, Lexical Similarity, EigenScore, and EigenScore-B. As shown in Table E, our DMP achieves significant efficiency gains on SciQ, and the mean AUROC even surpasses that of the baselines.

Appendix F Memory Cost

One limitation of our DMP is that it requires storing more cache compared to standard generation. To quantify the additional memory cost, we conduct experiments with LLaMA2-7B-Chat on TriviaQA using a batch size of 10. The baseline exhibits a peak reserved memory of 28,312 MB, whereas our DMP requires 29,034 MB, with an increase of 722MB.