E-BERT: Efficient-Yet-Effective Entity Embeddings for BERT

TL;DR

E-BERT introduces a method to incorporate factual entity knowledge into BERT by aligning entity vectors with BERT's wordpiece space, enhancing performance on QA, RC, and EL tasks without additional pretraining.

Contribution

The paper proposes a novel, efficient approach to embed entity knowledge into BERT by vector alignment, avoiding costly retraining.

Findings

E-BERT outperforms BERT on QA, RC, and EL tasks.

E-BERT reduces BERT's reliance on entity name surface forms.

The method requires no additional pretraining of BERT.

Abstract

We present a novel way of injecting factual knowledge about entities into the pretrained BERT model (Devlin et al., 2019): We align Wikipedia2Vec entity vectors (Yamada et al., 2016) with BERT's native wordpiece vector space and use the aligned entity vectors as if they were wordpiece vectors. The resulting entity-enhanced version of BERT (called E-BERT) is similar in spirit to ERNIE (Zhang et al., 2019) and KnowBert (Peters et al., 2019), but it requires no expensive further pretraining of the BERT encoder. We evaluate E-BERT on unsupervised question answering (QA), supervised relation classification (RC) and entity linking (EL). On all three tasks, E-BERT outperforms BERT and other baselines. We also show quantitatively that the original BERT model is overly reliant on the surface form of entity names (e.g., guessing that someone with an Italian-sounding name speaks Italian), and that E-BERT mitigates this problem.