Relation-Guided Pre-Training for Open-Domain Question Answering

TL;DR

This paper introduces RGPT-QA, a relation-guided pre-training framework that enhances open-domain question answering by addressing relation imbalance, leading to significant accuracy improvements especially on questions involving rare relations.

Contribution

The paper proposes a novel pre-training approach using a relational QA dataset to improve open-domain QA models' generalization on long-tail relations.

Findings

Achieved 2.2-6.3% accuracy improvements on benchmark datasets.

Significantly improved performance on questions with long-tail relations.

Demonstrated effectiveness of relation-guided pre-training in open-domain QA.

Abstract

Answering complex open-domain questions requires understanding the latent relations between involving entities. However, we found that the existing QA datasets are extremely imbalanced in some types of relations, which hurts the generalization performance over questions with long-tail relations. To remedy this problem, in this paper, we propose a Relation-Guided Pre-Training (RGPT-QA) framework. We first generate a relational QA dataset covering a wide range of relations from both the Wikidata triplets and Wikipedia hyperlinks. We then pre-train a QA model to infer the latent relations from the question, and then conduct extractive QA to get the target answer entity. We demonstrate that by pretraining with propoed RGPT-QA techique, the popular open-domain QA model, Dense Passage Retriever (DPR), achieves 2.2%, 2.4%, and 6.3% absolute improvement in Exact Match accuracy on Natural Questions, TriviaQA, and WebQuestions. Particularly, we show that RGPT-QA improves significantly on questions with long-tail relations