Pre-training Language Models with Deterministic Factual Knowledge

TL;DR

This paper introduces a pre-training approach for language models that emphasizes learning deterministic relationships in factual knowledge to improve robustness and performance in knowledge-intensive tasks.

Contribution

It proposes a novel pre-training method that leverages deterministic relationships using external knowledge bases to enhance factual knowledge robustness in PLMs.

Findings

Pre-trained models with the proposed method show improved robustness in factual knowledge extraction.

The approach benefits performance on question-answering and other knowledge-intensive tasks.

Models trained with deterministic relationships are less sensitive to prompt variations.

Abstract

Previous works show that Pre-trained Language Models (PLMs) can capture factual knowledge. However, some analyses reveal that PLMs fail to perform it robustly, e.g., being sensitive to the changes of prompts when extracting factual knowledge. To mitigate this issue, we propose to let PLMs learn the deterministic relationship between the remaining context and the masked content. The deterministic relationship ensures that the masked factual content can be deterministically inferable based on the existing clues in the context. That would provide more stable patterns for PLMs to capture factual knowledge than randomly masking. Two pre-training tasks are further introduced to motivate PLMs to rely on the deterministic relationship when filling masks. Specifically, we use an external Knowledge Base (KB) to identify deterministic relationships and continuously pre-train PLMs with the proposed methods. The factual knowledge probing experiments indicate that the continuously pre-trained PLMs achieve better robustness in factual knowledge capturing. Further experiments on question-answering datasets show that trying to learn a deterministic relationship with the proposed methods can also help other knowledge-intensive tasks.