Efficient Edge LLMs Deployment via HessianAware Quantization and CPU GPU Collaborative

TL;DR

This paper introduces a Hessian-Aware Quantization method and CPU-GPU collaborative inference strategy to efficiently deploy large language models on edge devices, reducing memory and latency while maintaining accuracy.

Contribution

It proposes a novel Hessian-Aware Quantization technique and expert-level collaborative inference mechanism for edge deployment of MoE models, addressing outlier-induced accuracy loss and resource constraints.

Findings

Achieves near full-precision accuracy with 8-bit quantization on large models.

Reduces GPU memory usage by approximately 60%.

Significantly improves inference latency on edge hardware.

Abstract

With the breakthrough progress of large language models (LLMs) in natural language processing and multimodal tasks, efficiently deploying them on resource-constrained edge devices has become a critical challenge. The Mixture of Experts (MoE) architecture enhances model capacity through sparse activation, but faces two major difficulties in practical deployment: (1) The presence of numerous outliers in activation distributions leads to severe degradation in quantization accuracy for both activations and weights, significantly impairing inference performance; (2) Under limited memory, efficient offloading and collaborative inference of expert modules struggle to balance latency and throughput. To address these issues, this paper proposes an efficient MoE edge deployment scheme based on Hessian-Aware Quantization (HAQ) and CPU-GPU collaborative inference. First, by introducing smoothed Hessian matrix quantization, we achieve joint 8-bit quantization of activations and weights, which significantly alleviates the accuracy loss caused by outliers while ensuring efficient implementation on mainstream hardware. Second, we design an expert-level collaborative offloading and inference mechanism, which, combined with expert activation path statistics, enables efficient deployment and scheduling of expert modules between CPU and GPU, greatly reducing memory footprint and inference latency. Extensive experiments validate the effectiveness of our method on mainstream large models such as the OPT series and Mixtral 8*7B: on datasets like Wikitext2 and C4, the inference accuracy of the low-bit quantized model approaches that of the full-precision model, while GPU memory usage is reduced by about 60%, and inference latency is significantly improved.