QuantX: A Framework for Hardware-Aware Quantization of Generative AI Workloads

TL;DR

QuantX introduces a hardware-aware quantization framework for large language and vision models, enabling 3-bit quantization with minimal performance loss and improved efficiency over existing methods.

Contribution

It provides a novel suite of quantization recipes tailored for hardware constraints, achieving near-original performance and integrating into popular frameworks.

Findings

QuantX quantizes models to 3 bits with less than 6% performance loss.

It outperforms recent state-of-the-art quantization techniques.

Integration into Llama.cpp demonstrates practical runtime efficiency.

Abstract

We present QuantX: a tailored suite of recipes for LLM and VLM quantization. It is capable of quantizing down to 3-bit resolutions with minimal loss in performance. The quantization strategies in QuantX take into account hardware-specific constraints to achieve efficient dequantization during inference ensuring flexible trade-off between runtime speed, memory requirement and model accuracy. Our results demonstrate that QuantX achieves performance within 6% of the unquantized model for LlaVa-v1.6 quantized down to 3-bits for multiple end user tasks and outperforms recently published state-of-the-art quantization techniques. We further integrate one particular technique from QuantX into the popular Llama.cpp framework and show its feasibility in terms of runtime compared to the mainstream quantization techniques from Llama.cpp. Lastly, this manuscript provides insights into the LLM quantization process that motivated the range of recipes and options that are incorporated in QuantX.