Relational World Knowledge Representation in Contextual Language Models: A Review

TL;DR

This review explores how deep contextual language models can internalize relational world knowledge, proposing a taxonomy based on supervision levels and highlighting current models, tasks, and future directions.

Contribution

It introduces an extensible taxonomy for knowledge representation in language models and reviews current models, evaluation tasks, and findings.

Findings

Language models can internalize relational knowledge with varying supervision levels

Current models demonstrate diverse capabilities in knowledge representation

Future research can enhance the integration of language models and knowledge bases

Abstract

Relational knowledge bases (KBs) are commonly used to represent world knowledge in machines. However, while advantageous for their high degree of precision and interpretability, KBs are usually organized according to manually-defined schemas, which limit their expressiveness and require significant human efforts to engineer and maintain. In this review, we take a natural language processing perspective to these limitations, examining how they may be addressed in part by training deep contextual language models (LMs) to internalize and express relational knowledge in more flexible forms. We propose to organize knowledge representation strategies in LMs by the level of KB supervision provided, from no KB supervision at all to entity- and relation-level supervision. Our contributions are threefold: (1) We provide a high-level, extensible taxonomy for knowledge representation in LMs; (2) Within our taxonomy, we highlight notable models, evaluation tasks, and findings, in order to provide an up-to-date review of current knowledge representation capabilities in LMs; and (3) We suggest future research directions that build upon the complementary aspects of LMs and KBs as knowledge representations.