Parallel Scaling Law for Language Models

TL;DR

This paper introduces parallel scaling (ParScale), a new inference-efficient method that increases model computation through parallel transformations, enabling similar or better performance with less memory and latency than traditional parameter scaling.

Contribution

The paper proposes ParScale, a novel parallel computation paradigm for language models, along with a new scaling law validated through large-scale experiments.

Findings

ParScale achieves comparable performance to parameter scaling with less memory and latency.

A new theoretical scaling law relates parallel streams to effective parameter scaling.

ParScale can convert pre-trained models into parallel versions with minimal additional training.

Abstract

It is commonly believed that scaling language models should commit a significant space or time cost, by increasing the parameters (parameter scaling) or output tokens (inference-time scaling). We introduce the third and more inference-efficient scaling paradigm: increasing the model's parallel computation during both training and inference time. We apply $P$ diverse and learnable transformations to the input, execute forward passes of the model in parallel, and dynamically aggregate the $P$ outputs. This method, namely parallel scaling (ParScale), scales parallel computation by reusing existing parameters and can be applied to any model structure, optimization procedure, data, or task. We theoretically propose a new scaling law and validate it through large-scale pre-training, which shows that a model with $P$ parallel streams is similar to scaling the parameters by $O(\log P)$ while showing superior inference efficiency. For example, ParScale can use up to 22$\times$ less memory increase and 6$\times$ less latency increase compared to parameter scaling that achieves the same performance improvement. It can also recycle an off-the-shelf pre-trained model into a parallelly scaled one by post-training on a small amount of tokens, further reducing the training budget. The new scaling law we discovered potentially facilitates the deployment of more powerful models in low-resource scenarios, and provides an alternative perspective for the role of computation in machine learning.