CoCoA: Confidence and Context-Aware Adaptive Decoding for Resolving Knowledge Conflicts in Large Language Models

TL;DR

CoCoA is a novel decoding algorithm that improves faithfulness in large language models by adaptively resolving knowledge conflicts using confidence and context measures, outperforming existing methods across multiple benchmarks.

Contribution

Introduces CoCoA, a conflict resolution decoding method that adaptively balances confidence and context, maintaining performance in low conflict scenarios and improving faithfulness in LLM outputs.

Findings

Achieves up to 9.2 points improvement in QA accuracy.

Enhances factuality in summarization and LFQA by up to 2.5 points.

Demonstrates superior sensitivity to conflict variations.

Abstract

Faithful generation in large language models (LLMs) is challenged by knowledge conflicts between parametric memory and external context. Existing contrastive decoding methods tuned specifically to handle conflict often lack adaptability and can degrade performance in low conflict settings. We introduce CoCoA (Confidence- and Context-Aware Adaptive Decoding), a novel token-level algorithm for principled conflict resolution and enhanced faithfulness. CoCoA resolves conflict by utilizing confidence-aware measures (entropy gap and contextual peakedness) and the generalized divergence between the parametric and contextual distributions. Crucially, CoCoA maintains strong performance even in low conflict settings. Extensive experiments across multiple LLMs on diverse Question Answering (QA), Summarization, and Long-Form Question Answering (LFQA) benchmarks demonstrate CoCoA's state-of-the-art performance over strong baselines like AdaCAD. It yields significant gains in QA accuracy, up to 9.2 points on average compared to the strong baseline AdaCAD, and improves factuality in summarization and LFQA by up to 2.5 points on average across key benchmarks. Additionally, it demonstrates superior sensitivity to conflict variations. CoCoA enables more informed, context-aware, and ultimately more faithful token generation.