Dynamic Attention-Guided Context Decoding for Mitigating Context Faithfulness Hallucinations in Large Language Models

TL;DR

This paper introduces DAGCD, a lightweight decoding framework that uses attention and uncertainty signals to reduce hallucinations in large language models, improving faithfulness and robustness without extra computational cost.

Contribution

The paper proposes DAGCD, a novel attention-guided decoding method that mitigates hallucinations in LLMs by leveraging attention distributions and uncertainty signals during decoding.

Findings

DAGCD significantly improves faithfulness in open-book QA tasks.

DAGCD enhances model robustness against hallucinations.

The method maintains computational efficiency during decoding.

Abstract

Large language models (LLMs) often exhibit Context Faithfulness Hallucinations, where outputs deviate from retrieved information due to incomplete context integration. Our analysis reveals a strong correlation between token-level uncertainty and hallucinations. We hypothesize that attention mechanisms inherently encode context utilization signals, supported by probing analysis. Based on these insights, we propose Dynamic Attention-Guided Context Decoding (DAGCD), a lightweight framework that leverages attention distributions and uncertainty signals in a single-pass decoding. Experiments on open-book QA datasets demonstrate DAGCD's effectiveness, yielding significant improvements in faithfulness and robustness while preserving computational efficiency.