A Neural Knowledge Language Model

TL;DR

The paper introduces a Neural Knowledge Language Model that integrates symbolic knowledge from knowledge graphs with RNNs to improve factual knowledge encoding and decoding in language generation.

Contribution

It presents a novel NKLM that combines symbolic knowledge with neural language modeling, enabling better factual knowledge handling.

Findings

Significantly improves language model performance on knowledge-related tasks.

Reduces the number of unknown words during generation.

Demonstrates effective integration of knowledge graphs with neural models.

Abstract

Current language models have a significant limitation in the ability to encode and decode factual knowledge. This is mainly because they acquire such knowledge from statistical co-occurrences although most of the knowledge words are rarely observed. In this paper, we propose a Neural Knowledge Language Model (NKLM) which combines symbolic knowledge provided by the knowledge graph with the RNN language model. By predicting whether the word to generate has an underlying fact or not, the model can generate such knowledge-related words by copying from the description of the predicted fact. In experiments, we show that the NKLM significantly improves the performance while generating a much smaller number of unknown words.