Dynamic Evaluation of Transformer Language Models

TL;DR

This paper combines Transformers and dynamic evaluation to enhance language model performance, achieving state-of-the-art results on multiple benchmarks by adapting models to recent sequence history.

Contribution

It introduces the application of dynamic evaluation to Transformer-XL models, significantly improving their performance on standard language modeling datasets.

Findings

Reduced bits/char on enwik8 from 0.99 to 0.94

Lowered perplexity on WikiText-103 from 18.3 to 16.4

Improved text8 bits/char from 1.08 to 1.04

Abstract

This research note combines two methods that have recently improved the state of the art in language modeling: Transformers and dynamic evaluation. Transformers use stacked layers of self-attention that allow them to capture long range dependencies in sequential data. Dynamic evaluation fits models to the recent sequence history, allowing them to assign higher probabilities to re-occurring sequential patterns. By applying dynamic evaluation to Transformer-XL models, we improve the state of the art on enwik8 from 0.99 to 0.94 bits/char, text8 from 1.08 to 1.04 bits/char, and WikiText-103 from 18.3 to 16.4 perplexity points.