LLM-PQ: Serving LLM on Heterogeneous Clusters with Phase-Aware Partition and Adaptive Quantization

TL;DR

This paper introduces LLM-PQ, a system that optimizes large language model serving on heterogeneous GPU clusters through adaptive quantization and phase-aware partitioning, significantly improving throughput and reducing costs.

Contribution

The paper presents a novel approach combining mixed-precision quantization and phase-aware partitioning tailored for heterogeneous GPU clusters, enhancing LLM inference efficiency.

Findings

Achieves up to 2.88x throughput improvement in inference.

Demonstrates effectiveness across 11 different production clusters.

Outperforms state-of-the-art methods in LLM serving efficiency.

Abstract

Recent breakthroughs in Large-scale language models (LLMs) have demonstrated impressive performance on various tasks. The immense sizes of LLMs have led to very high resource demand and cost for running the models. Though the models are largely served using uniform high-caliber GPUs nowadays, utilizing a heterogeneous cluster with a mix of available high- and low-capacity GPUs can potentially substantially reduce the serving cost. There is a lack of designs to support efficient LLM serving using a heterogeneous cluster, while the current solutions focus on model partition and uniform compression among homogeneous devices. This paper proposes LLM-PQ, a system that advocates adaptive model quantization and phase-aware partition to improve LLM serving efficiency on heterogeneous GPU clusters. We carefully decide on mixed-precision model quantization together with phase-aware model partition and micro-batch sizing in distributed LLM serving with an efficient algorithm, to greatly enhance inference throughput while fulfilling user-specified model quality targets. Extensive experiments on production inference workloads in 11 different clusters demonstrate that LLM-PQ achieves up to 2.88x (2.26x on average) throughput improvement in inference, showing great advantages over state-of-the-art works.