SmartQuant: CXL-based AI Model Store in Support of Runtime Configurable Weight Quantization

TL;DR

This paper proposes a CXL-based AI model store that enables runtime configurable weight quantization, improving inference efficiency, memory access speed, and energy efficiency for transformer models by leveraging hardware support and active CXL memory controllers.

Contribution

It introduces a novel CXL-based design that allows runtime configurable weight quantization to enhance AI inference performance and efficiency, filling a research gap in hardware exploitation.

Findings

Demonstrated improved inference efficiency on transformer models

Showed increased memory access speed and energy savings

Validated effectiveness through experimental results

Abstract

Recent studies have revealed that, during the inference on generative AI models such as transformer, the importance of different weights exhibits substantial context-dependent variations. This naturally manifests a promising potential of adaptively configuring weight quantization to improve the generative AI inference efficiency. Although configurable weight quantization can readily leverage the hardware support of variable-precision arithmetics in modern GPU and AI accelerators, little prior research has studied how one could exploit variable weight quantization to proportionally improve the AI model memory access speed and energy efficiency. Motivated by the rapidly maturing CXL ecosystem, this work develops a CXL-based design solution to fill this gap. The key is to allow CXL memory controllers play an active role in supporting and exploiting runtime configurable weight quantization. Using transformer as a representative generative AI model, we carried out experiments that well demonstrate the effectiveness of the proposed design solution.