SinkTrack: Attention Sink based Context Anchoring for Large Language Models

TL;DR

SinkTrack is a novel, training-free method that uses attention sink as an intrinsic anchor to improve context retention and reduce hallucinations in large language models during generation.

Contribution

It introduces SinkTrack, an advanced context anchoring technique leveraging attention sink, which is simple, effective, and applicable across various LLM architectures and scales.

Findings

Mitigates hallucination and context forgetting in LLMs.

Improves performance on textual and multi-modal tasks with significant gains.

Demonstrates robustness and generalizability across different models.

Abstract

Large language models (LLMs) suffer from hallucination and context forgetting. Prior studies suggest that attention drift is a primary cause of these problems, where LLMs' focus shifts towards newly generated tokens and away from the initial input context. To counteract this, we make use of a related, intrinsic characteristic of LLMs: attention sink -- the tendency to consistently allocate high attention to the very first token (i.e., <BOS>) of a sequence. Concretely, we propose an advanced context anchoring method, SinkTrack, which treats <BOS> as an information anchor and injects key contextual features (such as those derived from the input image or instruction) into its representation. As such, LLM remains anchored to the initial input context throughout the entire generation process. SinkTrack is training-free, plug-and-play, and introduces negligible inference overhead. Experiments demonstrate that SinkTrack mitigates hallucination and context forgetting across both textual (e.g., +21.6% on SQuAD2.0 with Llama3.1-8B-Instruct) and multi-modal (e.g., +22.8% on M3CoT with Qwen2.5-VL-7B-Instruct) tasks. Its consistent gains across different architectures and scales underscore the robustness and generalizability. We also analyze its underlying working mechanism from the perspective of information delivery. Our source code is available at https://github.com/67L1/SinkTrack.