Relational Memory Augmented Language Models

TL;DR

This paper introduces a memory-augmented language model that incorporates knowledge graphs as relation triples, enhancing text coherence and logical consistency in generation tasks.

Contribution

It presents a novel method to integrate knowledge graphs with autoregressive language models, improving performance and enabling causal interventions.

Findings

Improved perplexity and bits per character on benchmark datasets

Enhanced coherence and logical consistency in generated text

Complementary benefits when combined with token-based memory

Abstract

We present a memory-augmented approach to condition an autoregressive language model on a knowledge graph. We represent the graph as a collection of relation triples and retrieve relevant relations for a given context to improve text generation. Experiments on WikiText-103, WMT19, and enwik8 English datasets demonstrate that our approach produces a better language model in terms of perplexity and bits per character. We also show that relational memory improves coherence, is complementary to token-based memory, and enables causal interventions. Our model provides a simple yet effective way to combine an autoregressive language model with a knowledge graph for a more coherent and logical generation.