APSQ: Additive Partial Sum Quantization with Algorithm-Hardware Co-Design

TL;DR

This paper introduces APSQ, a novel quantization method that reduces energy consumption in DNN accelerators by efficiently compressing partial sums, with demonstrated benefits on NLP, CV, and large language models.

Contribution

APSQ integrates PSUM quantization into the model compression framework, enabling nearly lossless performance and significant energy savings across various neural network architectures.

Findings

Achieves nearly lossless quantization on NLP and CV tasks.

Reduces energy costs by 28-87% in DNN accelerators.

Demonstrates effectiveness on large language models like LLaMA2-7B.

Abstract

DNN accelerators, significantly advanced by model compression and specialized dataflow techniques, have marked considerable progress. However, the frequent access of high-precision partial sums (PSUMs) leads to excessive memory demands in architectures utilizing input/weight stationary dataflows. Traditional compression strategies have typically overlooked PSUM quantization, which may account for 69% of power consumption. This study introduces a novel Additive Partial Sum Quantization (APSQ) method, seamlessly integrating PSUM accumulation into the quantization framework. A grouping strategy that combines APSQ with PSUM quantization enhanced by a reconfigurable architecture is further proposed. The APSQ performs nearly lossless on NLP and CV tasks across BERT, Segformer, and EfficientViT models while compressing PSUMs to INT8. This leads to a notable reduction in energy costs by 28-87%. Extended experiments on LLaMA2-7B demonstrate the potential of APSQ for large language models. Code is available at https://github.com/Yonghao-Tan/APSQ.