When are 1.58 bits enough? A Bottom-up Exploration of BitNet Quantization

TL;DR

This paper investigates the effectiveness of 1.58-bit quantization across various neural network architectures, demonstrating that it can match or outperform traditional higher-precision models, thus enabling resource-efficient training and inference.

Contribution

It extends the application of 1.58-bit quantization beyond decoder-only language models to diverse architectures like MLPs, GNNs, and other transformer variants, showing consistent performance benefits.

Findings

1.58-bit training matches or exceeds standard models in multiple architectures.

Quantization enables resource-efficient training without sacrificing accuracy.

Results suggest broad applicability of low-bit quantization in neural networks.

Abstract

Contemporary machine learning models, such as language models, are powerful, but come with immense resource requirements both at training and inference time. It has been shown that decoder-only language models can be trained to a competitive state with ternary weights (1.58 bits per weight), facilitating efficient inference. Here, we start our exploration with non-transformer model architectures, investigating 1.58-bit training for multi-layer perceptrons and graph neural networks. Then, we explore 1.58-bit training in other transformer-based language models, namely encoder-only and encoder-decoder models. Our results show that in all of these settings, 1.58-bit training is on par with or sometimes even better than the standard 32/16-bit models.