On-The-Fly Information Retrieval Augmentation for Language Models

TL;DR

This paper explores augmenting pre-trained language models with on-the-fly information retrieval to improve performance, demonstrating significant perplexity reduction and validation on event co-reference tasks without retraining.

Contribution

It introduces a method of augmenting GPT-2 with dynamic IR, showing improvements in perplexity and task validation without additional training.

Findings

15% relative perplexity reduction on Gigaword

Effective IR augmentation on event co-reference

No re-training required for improvements

Abstract

Here we experiment with the use of information retrieval as an augmentation for pre-trained language models. The text corpus used in information retrieval can be viewed as form of episodic memory which grows over time. By augmenting GPT 2.0 with information retrieval we achieve a zero shot 15% relative reduction in perplexity on Gigaword corpus without any re-training. We also validate our IR augmentation on an event co-reference task.