Entropy-Based Decoding for Retrieval-Augmented Large Language Models

TL;DR

This paper introduces an entropy-guided decoding method for retrieval-augmented LLMs that improves factual accuracy by reducing distractibility from noisy external and internal knowledge sources.

Contribution

It presents a novel, training-free decoding approach combining entropy-based ensemble and contrastive decoding to enhance relevant information extraction in retrieval-augmented LLMs.

Findings

Outperforms existing methods on open-domain question answering datasets

Reduces distractibility caused by noisy knowledge sources

Improves factual accuracy of generated responses

Abstract

Augmenting Large Language Models (LLMs) with retrieved external knowledge has proven effective for improving the factual accuracy of generated responses. Despite their success, retrieval-augmented LLMs still face the distractibility issue, where the generated responses are negatively influenced by noise from both external and internal knowledge sources. In this paper, we introduce a novel, training-free decoding method guided by entropy considerations to mitigate this issue. Our approach utilizes entropy-based document-parallel ensemble decoding to prioritize low-entropy distributions from retrieved documents, thereby enhancing the extraction of relevant information of context. Additionally, it incorporates a contrastive decoding mechanism that contrasts the obtained low-entropy ensemble distribution with the high-entropy distribution derived from the model's internal knowledge across layers, which ensures a greater emphasis on reliable external information. Extensive experiments on open-domain question answering datasets demonstrate the superiority of our method.