SYMPHONY: Improving Memory Management for LLM Inference Workloads

TL;DR

SYMPHONY is a novel memory management system that enhances LLM inference workloads by dynamically migrating key-value caches, significantly increasing request throughput while maintaining low latency.

Contribution

It introduces a cache migration technique leveraging multi-turn workload hints to improve inference serving efficiency and scalability.

Findings

Handles over 8x more requests than baselines

Reduces redundant recomputation of tokens

Maintains similar latency profiles

Abstract

Large Language Models (LLMs) are increasingly being deployed in applications such as chatbots, code editors, and conversational agents. A key feature of LLMs is their ability to engage in multi-turn interactions with humans or external tools, enabling a wide range of tasks. Each new request in a multi-turn interaction depends on the intermediate state, specifically the key-value (K,V) caches, from previous requests in the ongoing interaction. Existing serving engines either recompute the K,V caches or offload them to main memory. Profiling reveals that recomputation can result in over 99% of processed tokens being redundant. On the other hand, offloading K,V caches from GPU memory makes inference serving stateful, leading to load imbalances across the cluster. To address these challenges, we developed SYMPHONY. SYMPHONY leverages the observation that multi-turn work loads provide additional hints that allow K,V caches to be migrated off the critical serving path. By utilizing these hints, SYMPHONY dynamically migrates K,V caches to enable finegrained scheduling of inference requests. Our experiments demonstrate that SYMPHONY can handle over 8x the number of requests compared to state-of-the-art baselines, with a similar latency profile.