Contextualized Representations Using Textual Encyclopedic Knowledge

TL;DR

This paper introduces a method to enhance text representations by dynamically integrating textual encyclopedic background knowledge, improving performance on question answering tasks, especially in factual reasoning scenarios.

Contribution

The paper proposes a novel approach to incorporate dynamic textual background knowledge into pretrained language models for improved factual reasoning in QA tasks.

Findings

Significant F1 score improvements on TriviaQA with background knowledge integration.

Consistent in-domain and out-of-domain gains across multiple QA datasets.

Enhanced performance through self-supervised pretraining with background-augmented inputs.

Abstract

We present a method to represent input texts by contextualizing them jointly with dynamically retrieved textual encyclopedic background knowledge from multiple documents. We apply our method to reading comprehension tasks by encoding questions and passages together with background sentences about the entities they mention. We show that integrating background knowledge from text is effective for tasks focusing on factual reasoning and allows direct reuse of powerful pretrained BERT-style encoders. Moreover, knowledge integration can be further improved with suitable pretraining via a self-supervised masked language model objective over words in background-augmented input text. On TriviaQA, our approach obtains improvements of 1.6 to 3.1 F1 over comparable RoBERTa models which do not integrate background knowledge dynamically. On MRQA, a large collection of diverse QA datasets, we see consistent gains in-domain along with large improvements out-of-domain on BioASQ (2.1 to 4.2 F1), TextbookQA (1.6 to 2.0 F1), and DuoRC (1.1 to 2.0 F1).