Options-Aware Dense Retrieval for Multiple-Choice query Answering

TL;DR

This paper introduces Options Aware Dense Retrieval (OADR), a novel fine-tuning method for dense retrieval models that improves evidence identification in long-context multiple-choice question answering tasks.

Contribution

OADR leverages query-options embeddings to better mimic oracle query embeddings, enhancing evidence retrieval for multiple-choice QA.

Findings

OADR outperforms existing baselines on QuALITY dataset.

Enhanced evidence retrieval leads to higher accuracy in multiple-choice QA.

Fine-tuning with query-options embeddings improves retrieval robustness.

Abstract

Long-context multiple-choice question answering tasks require robust reasoning over extensive text sources. Since most of the pre-trained transformer models are restricted to processing only a few hundred words at a time, successful completion of such tasks often relies on the identification of evidence spans, such as sentences, that provide supporting evidence for selecting the correct answer. Prior research in this domain has predominantly utilized pre-trained dense retrieval models, given the absence of supervision to fine-tune the retrieval process. This paper proposes a novel method called Options Aware Dense Retrieval (OADR) to address these challenges. ORDA uses an innovative approach to fine-tuning retrieval by leveraging query-options embeddings, which aim to mimic the embeddings of the oracle query (i.e., the query paired with the correct answer) for enhanced identification of supporting evidence. Through experiments conducted on the QuALITY benchmark dataset, we demonstrate that our proposed model surpasses existing baselines in terms of performance and accuracy.