Challenges in Deploying Long-Context Transformers: A Theoretical Peak Performance Analysis

TL;DR

This paper provides a theoretical framework for analyzing the efficiency challenges of deploying long-context transformers, focusing on the impact of large KV caches on computational cost, memory, and latency.

Contribution

It introduces a quantitative framework to analyze deployment challenges of long-context transformers, highlighting the KV cache as the main cost driver and guiding future cost reduction strategies.

Findings

Large KV cache causes increased computation and memory costs.

Prefilling and decoding with long contexts significantly increase latency.

Memory overflow leads to costly swapping, impacting performance.

Abstract

Transformer-based long context generative models power emerging AI applications like hour-long video understanding and project-level coding agent. Deploying long context transformers (e.g., 100K to 10M tokens) is prohibitively expensive compared to short context (e.g., 4K tokens) model variants. Reducing the cost of long-context transformers is becoming a pressing research and engineering challenge starting from the year of 2024. This work describes a concurrent programming framework for quantitatively analyzing the efficiency challenges in serving multiple long-context requests under limited size of GPU high-bandwidth memory (HBM) regime. We give a detailed analysis of how all additional computational costs, compared to 4K context, trace back to \textit{one single source: the large size of the KV cache}. We use a 34B GPT-3.5 level model of 50K context on A100 NVLink as a running example, and describe how its large KV cache causes four types of deployment challenges: (1) prefilling long inputs takes much longer compute time and GPU memory than short inputs; (2) after prefilling, the large KV cache residing on the GPU HBM substantially restricts the number of concurrent users being served; (3) during decoding, repeatedly reading the KV cache from HBM to SM largely increases latency; (4) when KV cache memory overflows, swapping it from HBM to DDR causes significant context switching latency. We use this framework to analyze existing works and identify possibilities of combining them to build end-to-end systems. Overall, this work offers a foundational framework for analyzing long context transformer deployment and identifies directions towards reducing the inference cost of 1M context to be as cheap as 4K.