SPOT: Knowledge-Enhanced Language Representations for Information Extraction

TL;DR

This paper introduces SPOT, a knowledge-enhanced language model that efficiently learns representations of entities and relationships from text spans, improving information extraction tasks with fewer parameters.

Contribution

The paper proposes a novel span-based pre-trained model that jointly encodes entities and relationships, addressing limitations of previous models in handling out-of-vocabulary entities and parameter efficiency.

Findings

Outperforms baselines in representing entities and relationships.

Fine-tuning surpasses RoBERTa in supervised information extraction tasks.

Achieves competitive results across various extraction benchmarks.

Abstract

Knowledge-enhanced pre-trained models for language representation have been shown to be more effective in knowledge base construction tasks (i.e.,~relation extraction) than language models such as BERT. These knowledge-enhanced language models incorporate knowledge into pre-training to generate representations of entities or relationships. However, existing methods typically represent each entity with a separate embedding. As a result, these methods struggle to represent out-of-vocabulary entities and a large amount of parameters, on top of their underlying token models (i.e.,~the transformer), must be used and the number of entities that can be handled is limited in practice due to memory constraints. Moreover, existing models still struggle to represent entities and relationships simultaneously. To address these problems, we propose a new pre-trained model that learns representations of both entities and relationships from token spans and span pairs in the text respectively. By encoding spans efficiently with span modules, our model can represent both entities and their relationships but requires fewer parameters than existing models. We pre-trained our model with the knowledge graph extracted from Wikipedia and test it on a broad range of supervised and unsupervised information extraction tasks. Results show that our model learns better representations for both entities and relationships than baselines, while in supervised settings, fine-tuning our model outperforms RoBERTa consistently and achieves competitive results on information extraction tasks.