Spectra 1.1: Scaling Laws and Efficient Inference for Ternary Language Models

TL;DR

This paper introduces Spectra-1.1, a suite of ternary language models trained on extensive data, and proposes novel packing schemes and a GPU kernel to significantly enhance inference efficiency, addressing memory bottlenecks in large language models.

Contribution

The paper presents Spectra-1.1, a large-scale suite of TriLMs, and introduces innovative 2-bit and 1.6-bit packing schemes along with TriRun GPU kernel for faster inference.

Findings

TriLMs benefit more from increased data than model scaling.

Proposed packing schemes accelerate CPU inference.

TriRun GPU kernel achieves up to 5x speedup.

Abstract

Large language models (LLMs) are increasingly used across research and industry applications, yet their inference efficiency remains a significant challenge. As the computational power of modern GPU architectures continuously improves, their memory bandwidth and capacity have not scaled proportionally, creating a critical bottleneck during inference. To address this, we investigate ternary language models (TriLMs) that employ quantization-aware training to significantly reduce memory requirements. We first analyze the scalability of TriLMs by conducting a scaling law analysis, revealing that TriLMs benefit more from increasing training data than from scaling model parameters. Based on this observation, we introduce Spectra-1.1, an open suite of TriLMs trained on up to 1.2 trillion tokens, demonstrating sustained performance gains at scale. Furthermore, to improve inference efficiency, we propose novel 2-bit and 1.6-bit packing schemes for ternary weights, which demonstrate accelerated inference across various CPU architectures. Also, building on the 2-bit packing, we develop a GPU kernel called TriRun that accelerates end-to-end model inference by up to 5 times compared to floating-point baselines. To encourage further exploration and development of TriLMs, we will release the Spectra-1.1 suite and TriRun inference kernels. Overall, our work lays the foundation for building and deploying efficient LLMs, providing a valuable resource for the research community.