The Journey Matters: Average Parameter Count over Pre-training Unifies Sparse and Dense Scaling Laws

TL;DR

This paper investigates optimal sparse pre-training configurations for large language models, introduces a unifying scaling law based on average parameter count, and demonstrates that sparse pre-training can match dense models' performance with reduced inference costs.

Contribution

It systematically explores sparse pre-training schedules, proposes a modified scaling law using average parameter count, and empirically validates its effectiveness across sparse and dense models.

Findings

Optimal pruning at 25%-75% of training compute

Modified scaling law accurately models evaluation loss

Sparse pre-training matches dense performance with fewer parameters

Abstract

Pruning eliminates unnecessary parameters in neural networks; it offers a promising solution to the growing computational demands of large language models (LLMs). While many focus on post-training pruning, sparse pre-training--which combines pruning and pre-training into a single phase--provides a simpler alternative. In this work, we present the first systematic exploration of optimal sparse pre-training configurations for LLMs through an examination of 80 unique pruning schedules across different sparsity levels and training durations. We find that initiating pruning at 25% of total training compute and concluding at 75% achieves near-optimal final evaluation loss. These findings provide valuable insights for efficient and effective sparse pre-training of LLMs. Furthermore, we propose a new scaling law that modifies the Chinchilla scaling law to use the average parameter count over pre-training. Through empirical and theoretical validation, we demonstrate that this modified scaling law accurately models evaluation loss for both sparsely and densely pre-trained LLMs, unifying scaling laws across pre-training paradigms. Our findings indicate that while sparse pre-training achieves the same final model quality as dense pre-training for equivalent compute budgets, it provides substantial benefits through reduced model size, enabling significant potential computational savings during inference.