A Review on Language Models as Knowledge Bases

TL;DR

This paper reviews how pretrained language models can serve as implicit knowledge bases, capturing information without human supervision, and discusses the necessary aspects for their effective use as KBs.

Contribution

It provides a comprehensive review of recent literature on using language models as knowledge bases and outlines key aspects for their effectiveness.

Findings

Language models encode significant knowledge in parameters

Probing LMs can reveal various types of knowledge

LM-based KBs require specific aspects for optimal performance

Abstract

Recently, there has been a surge of interest in the NLP community on the use of pretrained Language Models (LMs) as Knowledge Bases (KBs). Researchers have shown that LMs trained on a sufficiently large (web) corpus will encode a significant amount of knowledge implicitly in its parameters. The resulting LM can be probed for different kinds of knowledge and thus acting as a KB. This has a major advantage over traditional KBs in that this method requires no human supervision. In this paper, we present a set of aspects that we deem a LM should have to fully act as a KB, and review the recent literature with respect to those aspects.