Towards Low-bit Communication for Tensor Parallel LLM Inference

TL;DR

This paper proposes a quantization technique that significantly reduces communication bits in tensor parallel LLM inference, maintaining high performance while decreasing data transfer requirements.

Contribution

It introduces a novel quantization method leveraging consistent outliers to lower communication bits from 16 to 4.2 on average, enhancing efficiency without sacrificing accuracy.

Findings

Reduces communicated bits from 16 to 4.2 on average.

Maintains approximately 98% and 99.5% of original performance for Gemma 2 27B and Llama 2 13B.

Improves communication efficiency in large-scale tensor parallel LLM inference.

Abstract

Tensor parallelism provides an effective way to increase server large language model (LLM) inference efficiency despite adding an additional communication cost. However, as server LLMs continue to scale in size, they will need to be distributed across more devices, magnifying the communication cost. One way to approach this problem is with quantization, but current methods for LLMs tend to avoid quantizing the features that tensor parallelism needs to communicate. Taking advantage of consistent outliers in communicated features, we introduce a quantization method that reduces communicated values on average from 16 bits to 4.2 bits while preserving nearly all of the original performance. For instance, our method maintains around 98.0% and 99.5% of Gemma 2 27B's and Llama 2 13B's original performance, respectively, averaged across all tasks we evaluated on.