LoRE: Logit-Ranked Retriever Ensemble for Enhancing Open-Domain Question Answering

TL;DR

LoRE is a novel ensemble retrieval method that combines diverse retrievers and a logit-based ranking to improve open-domain question answering accuracy, especially for complex queries.

Contribution

It introduces a logit-ranked ensemble approach that mitigates positional bias and enhances answer relevance in retrieval-based QA systems.

Findings

LoRE outperforms existing methods on NarrativeQA and SQuAD.

Achieves significant improvements in EM, F1, and ROUGE-L scores.

Generates more relevant answers for complex questions.

Abstract

Retrieval-based question answering systems often suffer from positional bias, leading to suboptimal answer generation. We propose LoRE (Logit-Ranked Retriever Ensemble), a novel approach that improves answer accuracy and relevance by mitigating positional bias. LoRE employs an ensemble of diverse retrievers, such as BM25 and sentence transformers with FAISS indexing. A key innovation is a logit-based answer ranking algorithm that combines the logit scores from a large language model (LLM), with the retrieval ranks of the passages. Experimental results on NarrativeQA, SQuAD demonstrate that LoRE significantly outperforms existing retrieval-based methods in terms of exact match and F1 scores. On SQuAD, LoRE achieves 14.5\%, 22.83\%, and 14.95\% improvements over the baselines for ROUGE-L, EM, and F1, respectively. Qualitatively, LoRE generates more relevant and accurate answers, especially for complex queries.