Improving Generated and Retrieved Knowledge Combination Through Zero-shot Generation

TL;DR

This paper introduces BRMGR, an unsupervised re-ranking framework that effectively combines retrieved and generated knowledge for open-domain QA, improving performance across multiple datasets.

Contribution

The paper proposes a novel unsupervised re-ranking approach for merging generated and retrieved knowledge, addressing the lack of labels in open-domain QA.

Findings

Improved accuracy on NQ and WebQ datasets (+1.7 and +1.6 points)

Achieved competitive results on TriviaQA dataset

Demonstrated equivalence to bipartite matching loss in knowledge pairing

Abstract

Open-domain Question Answering (QA) has garnered substantial interest by combining the advantages of faithfully retrieved passages and relevant passages generated through Large Language Models (LLMs). However, there is a lack of definitive labels available to pair these sources of knowledge. In order to address this issue, we propose an unsupervised and simple framework called Bi-Reranking for Merging Generated and Retrieved Knowledge (BRMGR), which utilizes re-ranking methods for both retrieved passages and LLM-generated passages. We pair the two types of passages using two separate re-ranking methods and then combine them through greedy matching. We demonstrate that BRMGR is equivalent to employing a bipartite matching loss when assigning each retrieved passage with a corresponding LLM-generated passage. The application of our model yielded experimental results from three datasets, improving their performance by +1.7 and +1.6 on NQ and WebQ datasets, respectively, and obtaining comparable result on TriviaQA dataset when compared to competitive baselines.