Precision-Scalable Microscaling Datapaths with Optimized Reduction Tree for Efficient NPU Integration

TL;DR

This paper introduces a hybrid precision-scalable reduction tree for MX MACs, enabling efficient mixed-precision accumulation in NPUs, and demonstrates significant energy efficiency improvements in an integrated system.

Contribution

It proposes a novel hybrid reduction tree for MX MACs and integrates it into a state-of-the-art NPU platform, enhancing energy efficiency and throughput for next-generation neural processing.

Findings

Achieves up to 4065 GOPS/W energy efficiency.

Supports multiple MX precisions with high throughput.

Demonstrates improved system-level performance over state-of-the-art.

Abstract

Emerging continual learning applications necessitate next-generation neural processing unit (NPU) platforms to support both training and inference operations. The promising Microscaling (MX) standard enables narrow bit-widths for inference and large dynamic ranges for training. However, existing MX multiply-accumulate (MAC) designs face a critical trade-off: integer accumulation requires expensive conversions from narrow floating-point products, while FP32 accumulation suffers from quantization losses and costly normalization. To address these limitations, we propose a hybrid precision-scalable reduction tree for MX MACs that combines the benefits of both approaches, enabling efficient mixed-precision accumulation with controlled accuracy relaxation. Moreover, we integrate an 8x8 array of these MACs into the state-of-the-art (SotA) NPU integration platform, SNAX, to provide efficient control and data transfer to our optimized precision-scalable MX datapath. We evaluate our design both on MAC and system level and compare it to the SotA. Our integrated system achieves an energy efficiency of 657, 1438-1675, and 4065 GOPS/W, respectively, for MXINT8, MXFP8/6, and MXFP4, with a throughput of 64, 256, and 512 GOPS.