TP-Aware Dequantization

TL;DR

This paper introduces a TP-aware dequantization method that significantly accelerates large language model inference by optimizing GPU memory access and reducing communication, achieving up to 1.81x speedup.

Contribution

It presents a novel inference scheme that addresses quantization kernel limitations with tensor parallelism, improving deployment speed of large models.

Findings

Up to 1.81x speedup on Llama-70B

Up to 1.78x speedup on IBM WatsonX Granite-20B

Effective across various tensor parallel settings

Abstract

In this paper, we present a novel method that reduces model inference latency during distributed deployment of Large Language Models (LLMs). Our contribution is an optimized inference deployment scheme that address the current limitations of state-of-the-art quantization kernels when used in conjunction with Tensor Parallel (TP). Our method preserves data locality in GPU memory access patterns and exploits a priori knowledge of TP to reduce global communication. We demonstrate an up to 1.81x speedup over existing methods for Llama-70B and up to 1.78x speedup for IBM WatsonX's Granite-20B MLP layer problem sizes on A100 and H100 NVIDIA DGX Systems for a variety of TP settings.