W4A16 Mixed-Precision Matrix Multiplication on Decoupled Architecture: Kernel Design and Memory Bottleneck Analysis for Ascend NPUs

TL;DR

This paper introduces a practical W4A16 matrix multiplication kernel for Huawei's Ascend 910 NPU, optimizing mixed-precision LLM deployment by addressing architecture-specific challenges and memory bottlenecks.

Contribution

It presents the first tailored W4A16 kernel for Ascend 910, leveraging vector and cube cores, and analyzes performance bottlenecks for efficient quantized LLM deployment.

Findings

Achieves 1.01x to 1.74x speedup over data-parallel methods.

Memory transfer, not dequantization, is the main bottleneck.

Maximum speedup of 1.48x over native FP16 in PyTorch.

Abstract

As Large Language Models (LLMs) scale, weight-only quantization (W4A16: 4-bit weights, 16-bit activations) becomes critical for reducing memory footprint with minimal accuracy loss. However, its efficient deployment on Huawei's Ascend 910 Neural Processing Unit (NPU) is challenging due to limited native mixed-precision support and the accelerator's decoupled compute architecture. To enable quantization on such architecture, we present the first practical W4A16 matrix multiplication kernel tailored for the Ascend 910 NPU. Our design leverages vector cores for on-the-fly INT4-to-FP16 dequantization, cube cores for high-throughput GEMM, and Split-K parallelization to mitigate memory latency. Performance evaluations across diverse matrix shapes and batch sizes show our method outperforms data-parallel approaches when K >> N, a typical scenario in LLM decoding. Specially, our method can achieve a speedup ranging from 1.01x to 1.74x. In addition, our profile reveals the primary bottleneck is not dequantization compution itself, but extra global memory transfer for the weight, making W4A16 only reaching a maximum speedup of 1.48x over native FP16xFP16 matrix multiplication in PyTorch. In the long run, our method lays a solid foundation and provides insightful views for the efficient deployment of quantized large language models on various domain-specific accelerators.