Parameters vs FLOPs: Scaling Laws for Optimal Sparsity for Mixture-of-Experts Language Models

TL;DR

This paper investigates how the sparsity level in Mixture-of-Experts language models affects their performance and training efficiency, revealing optimal sparsity points that enhance scaling laws and model capabilities.

Contribution

It provides new insights into the relationship between sparsity, parameters, and FLOPs in MoEs, identifying optimal sparsity levels for improved efficiency and performance.

Findings

Optimal sparsity improves training efficiency.

Sparsity levels affect downstream task performance.

Scaling laws are influenced by sparsity constraints.

Abstract

Scaling the capacity of language models has consistently proven to be a reliable approach for improving performance and unlocking new capabilities. Capacity can be primarily defined by two dimensions: the number of model parameters and the compute per example. While scaling typically involves increasing both, the precise interplay between these factors and their combined contribution to overall capacity remains not fully understood. We explore this relationship in the context of sparse Mixture-of-Experts (MoEs), which allow scaling the number of parameters without proportionally increasing the FLOPs per example. We investigate how varying the sparsity level, i.e., the fraction of inactive parameters, impacts model's performance during pretraining and downstream few-shot evaluation. We find that under different constraints (e.g., parameter size and total training compute), there is an optimal level of sparsity that improves both training efficiency and model performance. These results provide a better understanding of the impact of sparsity in scaling laws for MoEs and complement existing works in this area, offering insights for designing more efficient architectures.