Fast NF4 Dequantization Kernels for Large Language Model Inference

TL;DR

This paper introduces a lightweight shared memory optimization for NF4 dequantization in LLM inference, significantly boosting kernel speed and end-to-end performance on NVIDIA GPUs.

Contribution

It presents a novel shared memory-based dequantization kernel that improves speed and efficiency, compatible with existing ecosystems and minimal engineering effort.

Findings

Achieved 2.0--2.2× kernel speedup over BitsAndBytes

Up to 1.54× end-to-end performance improvement

Reduced shared memory usage to 64 bytes per thread block

Abstract

Large language models (LLMs) have grown beyond the memory capacity of single GPU devices, necessitating quantization techniques for practical deployment. While NF4 (4-bit NormalFloat) quantization enables 4$\times$ memory reduction, inference on current NVIDIA GPUs (e.g., Ampere A100) requires expensive dequantization back to FP16 format, creating a critical performance bottleneck. This paper presents a lightweight shared memory optimization that addresses this gap through principled memory hierarchy exploitation while maintaining full ecosystem compatibility. We compare our technique against the open-source BitsAndBytes implementation, achieving 2.0--2.2$\times$ kernel speedup across three models (Gemma 27B, Qwen3 32B, and Llama3.3 70B) and up to 1.54$\times$ end-to-end improvement by leveraging the 12--15$\times$ latency advantage of shared memory over global memory access. Our optimization reduces instruction counts through simplified indexing logic while using only 64 bytes of shared memory per thread block, demonstrating that lightweight optimizations can deliver substantial performance gains with minimal engineering effort. This work provides a plug-and-play solution for the HuggingFace ecosystem that democratizes access to advanced models on existing GPU infrastructure.