Memory-based Language Models: An Efficient, Explainable, and Eco-friendly Approach to Large Language Modeling

TL;DR

This paper introduces memory-based language modeling as an efficient, transparent, and eco-friendly alternative to neural networks, achieving scalable performance with low environmental impact and high interpretability.

Contribution

It presents OLIFANT, a memory-based language model that outperforms traditional neural models in efficiency, transparency, and ecological footprint, with detailed comparisons to GPT-2 and GPT-Neo.

Findings

OLIFANT achieves competitive next-token prediction accuracy.

It has a significantly lower ecological footprint during training and inference.

The model offers high transparency and interpretability.

Abstract

We present memory-based language modeling as an efficient, eco-friendly alternative to deep neural network-based language modeling. It offers log-linearly scalable next-token prediction performance and strong memorization capabilities. Implementing fast approximations of k-nearest neighbor classification, memory-based language modeling leaves a relatively small ecological footprint both in training and in inference mode, as it relies fully on CPUs and attains low token latencies. Its internal workings are simple and fully transparent. We compare our implementation of memory-based language modeling, OLIFANT, with GPT-2 and GPT-Neo on next-token prediction accuracy, estimated emissions and speeds, and offer some deeper analyses of the model.