Pointwise Mutual Information as a Performance Gauge for Retrieval-Augmented Generation

TL;DR

This paper introduces pointwise mutual information as a novel, answer-agnostic metric to gauge and improve retrieval-augmented language model performance, demonstrated through experiments on question-answering tasks.

Contribution

It proposes using pointwise mutual information as a performance gauge and introduces methods to enhance prompt selection for retrieval-augmented generation.

Findings

Empirical correlation between answer accuracy and pointwise mutual information.

Effective prompt construction methods based on mutual information.

Improved question-answering performance using the proposed gauge.

Abstract

Recent work suggests that large language models enhanced with retrieval-augmented generation are easily influenced by the order, in which the retrieved documents are presented to the model when solving tasks such as question answering (QA). However, there is no method to date that exploits this phenomenon to improve generation. We fill this gap. In this study, we show that the pointwise mutual information between a context and a question is an effective gauge for language model performance. Importantly, this gauge does not depend on knowing the answer to the question a priori. Through experiments on two question-answering datasets and a variety of large language models, we find evidence for an empirical correlation between answer accuracy and pointwise mutual information. Additionally, we propose two methods that use the pointwise mutual information between a document and a question as a gauge for selecting and constructing prompts that lead to better performance, whose effectiveness we demonstrate through experimentation.