Spectra: Surprising Effectiveness of Pretraining Ternary Language Models at Scale

TL;DR

This paper demonstrates that ternary language models (TriLMs) trained at scale can outperform traditional floating-point and quantized models of similar bit-widths, offering a promising path for more efficient large language models.

Contribution

It introduces the Spectra LLM suite, the first open collection of models across multiple bit-widths, and shows that TriLMs outperform quantized and float models at large scales, challenging existing assumptions.

Findings

TriLMs outperform QuantLMs and FloatLMs at scales over one billion parameters.

The 3.9B TriLM matches the performance of the 3.9B FloatLM despite fewer bits.

Open release of 500+ checkpoints facilitates further research.

Abstract

Rapid advancements in GPU computational power has outpaced memory capacity and bandwidth growth, creating bottlenecks in Large Language Model (LLM) inference. Post-training quantization is the leading method for addressing memory-related bottlenecks in LLM inference, but it suffers from significant performance degradation below 4-bit precision. This paper addresses these challenges by investigating the pretraining of low-bitwidth models specifically Ternary Language Models (TriLMs) as an alternative to traditional floating-point models (FloatLMs) and their post-training quantized versions (QuantLMs). We present Spectra LLM suite, the first open suite of LLMs spanning multiple bit-widths, including FloatLMs, QuantLMs, and TriLMs, ranging from 99M to 3.9B parameters trained on 300B tokens. Our comprehensive evaluation demonstrates that TriLMs offer superior scaling behavior in terms of model size (in bits). Surprisingly, at scales exceeding one billion parameters, TriLMs consistently outperform their QuantLM and FloatLM counterparts for a given bit size across various benchmarks. Notably, the 3.9B parameter TriLM matches the performance of the FloatLM 3.9B across all benchmarks, despite having fewer bits than FloatLM 830M. Overall, this research provides valuable insights into the feasibility and scalability of low-bitwidth language models, paving the way for the development of more efficient LLMs. To enhance understanding of low-bitwidth models, we are releasing 500+ intermediate checkpoints of the Spectra suite at https://github.com/NolanoOrg/SpectraSuite.