Communication Compression for Tensor Parallel LLM Inference

TL;DR

This paper proposes a communication compression method using fine-grained quantization for tensor parallel LLM inference, significantly reducing latency and inter-accelerator communication without notable performance loss.

Contribution

It introduces a novel quantization-based compression technique specifically designed for tensor parallel LLM inference to reduce communication overhead.

Findings

Achieves 3.5-4.5x compression of activations.

Reduces time-to-first-token by up to 2x.

Maintains negligible model performance degradation.

Abstract

Large Language Models (LLMs) have pushed the frontier of artificial intelligence but are comprised of hundreds of billions of parameters and operations. For faster inference latency, LLMs are deployed on multiple hardware accelerators through various Model Parallelism strategies. Our paper looks into the details on one such strategy - Tensor Parallel - and proposes to reduce latency by compressing inter-accelerator communication. We leverage fine grained quantization techniques to compress selected activations by 3.5 - 4.5x. Our proposed method leads up to 2x reduction of time-to-first-token (TTFT) with negligible model performance degradation.