Spelling Bee Embeddings for Language Modeling

TL;DR

This paper proposes spelling bee embeddings that incorporate spelling information into token embeddings, leading to improved language model performance and efficiency across various benchmarks and model sizes.

Contribution

It introduces a novel embedding modification that infuses spelling information, enhancing model performance and reducing data and compute requirements.

Findings

Improved performance on spelling and standard benchmarks.

Equivalent to 8% less compute and data for the same test loss.

Effective across models from 40M to 800M parameters.

Abstract

We introduce a simple modification to the embedding layer. The key change is to infuse token embeddings with information about their spelling. Models trained with these embeddings improve not only on spelling, but also across standard benchmarks. We conduct scaling studies for models with 40M to 800M parameters, which suggest that the improvements are equivalent to needing about 8% less compute and data to achieve the same test loss.