ContextFocus: Activation Steering for Contextual Faithfulness in Large Language Models

TL;DR

ContextFocus is a lightweight, efficient activation steering method that enhances the contextual faithfulness of large language models without requiring finetuning, especially in knowledge-conflict scenarios.

Contribution

It introduces a novel activation steering approach that improves contextual faithfulness in LLMs without finetuning and with minimal inference overhead.

Findings

Significantly improves contextual faithfulness on ConFiQA benchmark.

Remains effective on larger models and complements prompting strategies.

Requires no model finetuning and has minimal inference overhead.

Abstract

Large Language Models (LLMs) encode vast amounts of parametric knowledge during pre-training. As world knowledge evolves, effective deployment increasingly depends on their ability to faithfully follow externally retrieved context. When such evidence conflicts with the model's internal knowledge, LLMs often default to memorized facts, producing unfaithful outputs. In this work, we introduce ContextFocus, a lightweight activation steering approach that improves context faithfulness in such knowledge-conflict settings while preserving fluency and efficiency. Unlike prior approaches, our solution requires no model finetuning and incurs minimal inference-time overhead, making it highly efficient. We evaluate ContextFocus on the ConFiQA benchmark, comparing it against strong baselines including ContextDPO, COIECD, and prompting-based methods. Furthermore, we show that our method is complementary to prompting strategies and remains effective on larger models. Extensive experiments show that ContextFocus significantly improves contextual-faithfulness. Our results highlight the effectiveness, robustness, and efficiency of ContextFocus in improving contextual-faithfulness of LLM outputs.