BitMoD: Bit-serial Mixture-of-Datatype LLM Acceleration

TL;DR

BitMoD introduces a co-designed algorithm-hardware approach that enables low-precision quantization of LLM weights and efficient acceleration, significantly improving speed and maintaining high accuracy across various tasks.

Contribution

The paper presents a novel data type adaptation and a bit-serial hardware design for low-precision LLM acceleration, achieving superior performance and accuracy.

Findings

Quantizes LLM weights to 4 bits with <0.5% accuracy loss on discriminative tasks.

Quantizes LLM weights to 3 bits with better perplexity on generative tasks.

Achieves 1.69x and 1.48x speedups over prior accelerators ANT and OliVe.

Abstract

Large language models (LLMs) have demonstrated remarkable performance across various machine learning tasks. Yet the substantial memory footprint of LLMs significantly hinders their deployment. In this paper, we improve the accessibility of LLMs through BitMoD, an algorithm-hardware co-design solution that enables efficient LLM acceleration at low weight precision. On the algorithm side, BitMoD introduces fine-grained data type adaptation that uses a different numerical data type to quantize a group of (e.g., 128) weights. Through the careful design of these new data types, BitMoD is able to quantize LLM weights to very low precision (e.g., 4 bits and 3 bits) while maintaining high accuracy. On the hardware side, BitMoD employs a bit-serial processing element to easily support multiple numerical precisions and data types; our hardware design includes two key innovations: First, it employs a unified representation to process different weight data types, thus reducing the hardware cost. Second, it adopts a bit-serial dequantization unit to rescale the per-group partial sum with minimal hardware overhead. Our evaluation on six representative LLMs demonstrates that BitMoD significantly outperforms state-of-the-art LLM quantization and acceleration methods. For discriminative tasks, BitMoD can quantize LLM weights to 4-bit with $<\!0.5\%$ accuracy loss on average. For generative tasks, BitMoD is able to quantize LLM weights to 3-bit while achieving better perplexity than prior LLM quantization scheme. Combining the superior model performance with an efficient accelerator design, BitMoD achieves an average of $1.69\times$ and $1.48\times$ speedups compared to prior LLM accelerators ANT and OliVe, respectively.