BWTA: Accurate and Efficient Binarized Transformer by Algorithm-Hardware Co-design

TL;DR

BWTA introduces a novel ultra low-bit quantization scheme with algorithm-hardware co-design, achieving near full-precision accuracy and significant speedups for Transformer models on GPUs.

Contribution

The paper proposes a new binarization and ternarization scheme with specialized training and inference methods, enabling efficient ultra low-bit Transformer inference without accuracy loss.

Findings

Approaches full-precision performance on BERT with minimal accuracy drop.

Achieves 16-24x speedup over FP16 on NVIDIA GPUs.

Delivers 216-330 tokens/sec end-to-end speedup with lower memory footprint.

Abstract

Ultra low-bit quantization brings substantial efficiency for Transformer-based models, but the accuracy degradation and limited GPU support hinder its wide usage. In this paper, we analyze zero-point distortion in binarization and propose a Binary Weights & Ternary Activations (BWTA) quantization scheme, which projects tiny values to zero and preserves the accuracy of extremely low-bit models. For training, we propose Smooth Multi-Stage Quantization, combining a Levelwise Degradation Strategy and a Magnitude-Alignment Projection Factor to enable stable and fast convergence. For inference, we develop a BWTA MatMul CUDA kernel with instruction-level parallel bit-packing and comprehensive binary/ternary MatMul implementations for both linear and attention operators, allowing seamless integration across Transformer architectures. Experiments show that BWTA approaches full-precision performance for BERT, with an average 3.5% drop on GLUE and less than 2% drop on five tasks, and achieves comparable perplexity and accuracy for LLMs. In efficiency, it delivers 16 to 24 times kernel-level speedup over FP16 on NVIDIA GPUs, and 216 to 330 tokens/s end-to-end prefill speedup with lower memory footprint on LLMs. As an algorithm-hardware co-design, BWTA demonstrates practical, low-latency ultra-low-bit inference without sacrificing model quality.