A Simple but Effective Pluggable Entity Lookup Table for Pre-trained Language Models

TL;DR

This paper introduces PELT, a simple, pluggable entity lookup table that enhances pre-trained language models by efficiently incorporating rich entity knowledge, especially for rare and out-of-domain entities.

Contribution

The paper proposes PELT, a lightweight, effective method for infusing entity knowledge into PLMs with minimal pre-computation and broad domain adaptability.

Findings

PELT improves entity knowledge recall in PLMs.

PELT requires only 0.2%-5% pre-computation.

PELT effectively transfers knowledge across different domains.

Abstract

Pre-trained language models (PLMs) cannot well recall rich factual knowledge of entities exhibited in large-scale corpora, especially those rare entities. In this paper, we propose to build a simple but effective Pluggable Entity Lookup Table (PELT) on demand by aggregating the entity's output representations of multiple occurrences in the corpora. PELT can be compatibly plugged as inputs to infuse supplemental entity knowledge into PLMs. Compared to previous knowledge-enhanced PLMs, PELT only requires 0.2%-5% pre-computation with capability of acquiring knowledge from out-of-domain corpora for domain adaptation scenario. The experiments on knowledge-related tasks demonstrate that our method, PELT, can flexibly and effectively transfer entity knowledge from related corpora into PLMs with different architectures.