SimpleBooks: Long-term dependency book dataset with simplified English vocabulary for word-level language modeling

TL;DR

SimpleBooks is a new dataset of 92 million tokens with simplified English vocabulary, designed to facilitate efficient long-term language modeling and architectural research by mimicking properties of larger datasets.

Contribution

It introduces a small, high-frequency vocabulary dataset that matches large datasets in token count, aiding faster and more effective language model training and architecture search.

Findings

Contains 92M tokens, comparable to WikiText-103.

Has a vocabulary of 98K words, much smaller than WikiText-103.

Designed to replicate properties of larger datasets for research convenience.

Abstract

With language modeling becoming the popular base task for unsupervised representation learning in Natural Language Processing, it is important to come up with new architectures and techniques for faster and better training of language models. However, due to a peculiarity of languages -- the larger the dataset, the higher the average number of times a word appears in that dataset -- datasets of different sizes have very different properties. Architectures performing well on small datasets might not perform well on larger ones. For example, LSTM models perform well on WikiText-2 but poorly on WikiText-103, while Transformer models perform well on WikiText-103 but not on WikiText-2. For setups like architectural search, this is a challenge since it is prohibitively costly to run a search on the full dataset but it is not indicative to experiment on smaller ones. In this paper, we introduce SimpleBooks, a small dataset with the average word frequency as high as that of much larger ones. Created from 1,573 Gutenberg books with the highest ratio of word-level book length to vocabulary size, SimpleBooks contains 92M word-level tokens, on par with WikiText-103 (103M tokens), but has the vocabulary of 98K, a third of WikiText-103's. SimpleBooks can be downloaded from https://dldata-public.s3.us-east-2.amazonaws.com/simplebooks.zip.