Scaling Embedding Layers in Language Models

TL;DR

The paper introduces SCONE, a scalable n-gram embedding method that improves language model performance by adding contextualized embeddings without increasing inference costs, enabling effective scaling of model size and embeddings.

Contribution

SCONE provides a novel approach to extend embedding layers with n-gram embeddings, allowing scalable model improvements while maintaining low inference latency.

Findings

Scaling n-gram embeddings improves model performance.

Model with 1B parameters outperforms 1.9B baseline.

Inference cost remains low despite scaling.

Abstract

We propose $SCONE$ ($S$calable, $C$ontextualized, $O$ffloaded, $N$-gram $E$mbedding), a new method for extending input embedding layers to enhance language model performance. To avoid increased decoding costs, $SCONE$ retains the original vocabulary while introducing embeddings for a set of frequent n-grams. These embeddings provide contextualized representation for each input token and are learned with a separate model during training. After training, embeddings are precomputed and stored in off-accelerator memory; during inference, querying them has minimal impact on latency due to the low complexity of embedding lookups. $SCONE$ enables two new scaling strategies: increasing the number of n-gram embeddings and scaling the model used to learn them, both while maintaining fixed accelerator usage during inference (in terms of FLOPS and memory). We show that scaling both aspects enables a model with 1B accelerator-resident parameters to outperform a 1.9B-parameter baseline across diverse corpora, while using only about half the FLOPS and accelerator memory during inference.