FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU

TL;DR

FlexGen introduces a novel approach for high-throughput large language model inference on a single GPU by optimizing memory and computation through tensor storage patterns, weight compression, and resource aggregation.

Contribution

It presents FlexGen, a flexible inference engine that enables high throughput of LLMs on limited hardware by solving a linear programming problem for tensor management and compressing weights to 4 bits.

Findings

Achieves 1 token/sec throughput for OPT-175B on a 16GB GPU.

Enables benchmarking of 30B models within 21 hours on a single GPU.

Significantly outperforms existing offloading systems in throughput.

Abstract

The high computational and memory requirements of large language model (LLM) inference make it feasible only with multiple high-end accelerators. Motivated by the emerging demand for latency-insensitive tasks with batched processing, this paper initiates the study of high-throughput LLM inference using limited resources, such as a single commodity GPU. We present FlexGen, a high-throughput generation engine for running LLMs with limited GPU memory. FlexGen can be flexibly configured under various hardware resource constraints by aggregating memory and computation from the GPU, CPU, and disk. By solving a linear programming problem, it searches for efficient patterns to store and access tensors. FlexGen further compresses the weights and the attention cache to 4 bits with negligible accuracy loss. These techniques enable FlexGen to have a larger space of batch size choices and thus significantly increase maximum throughput. As a result, when running OPT-175B on a single 16GB GPU, FlexGen achieves significantly higher throughput compared to state-of-the-art offloading systems, reaching a generation throughput of 1 token/s for the first time with an effective batch size of 144. On the HELM benchmark, FlexGen can benchmark a 30B model with a 16GB GPU on 7 representative sub-scenarios in 21 hours. The code is available at https://github.com/FMInference/FlexGen