GLaM: Efficient Scaling of Language Models with Mixture-of-Experts

TL;DR

GLaM is a sparsely activated mixture-of-experts language model with 1.2 trillion parameters that achieves superior zero-shot and one-shot NLP task performance while reducing training energy and inference computation compared to GPT-3.

Contribution

This paper introduces GLaM, a scalable mixture-of-experts language model that significantly reduces training costs and energy consumption while improving performance over dense models like GPT-3.

Findings

GLaM has 1.2 trillion parameters, about 7 times GPT-3.

GLaM uses only one-third of GPT-3's training energy.

GLaM requires half the inference FLOPs of GPT-3 and outperforms it on 29 NLP tasks.

Abstract

Scaling language models with more data, compute and parameters has driven significant progress in natural language processing. For example, thanks to scaling, GPT-3 was able to achieve strong results on in-context learning tasks. However, training these large dense models requires significant amounts of computing resources. In this paper, we propose and develop a family of language models named GLaM (Generalist Language Model), which uses a sparsely activated mixture-of-experts architecture to scale the model capacity while also incurring substantially less training cost compared to dense variants. The largest GLaM has 1.2 trillion parameters, which is approximately 7x larger than GPT-3. It consumes only 1/3 of the energy used to train GPT-3 and requires half of the computation flops for inference, while still achieving better overall zero-shot and one-shot performance across 29 NLP tasks.