Memory Offloading for Large Language Model Inference with Latency SLO Guarantees

TL;DR

This paper introduces Select-N, a latency-aware memory offloading system for large language models that optimizes host memory usage while ensuring latency SLOs, significantly improving inference throughput.

Contribution

Select-N uniquely exploits deterministic decoder layer times in LLMs to balance latency guarantees with memory utilization through an adaptive offloading interval.

Findings

Select-N consistently meets latency SLOs.

It improves inference throughput by 1.85 times.

Maximizes host memory use without violating latency constraints.

Abstract

Offloading large language models (LLMs) state to host memory during inference promises to reduce operational costs by supporting larger models, longer inputs, and larger batch sizes. However, the design of existing memory offloading mechanisms does not take latency service-level objectives (SLOs) into consideration. As a result, they either lead to frequent SLO violations or underutilize host memory, thereby incurring economic loss and thus defeating the purpose of memory offloading. This paper presents Select-N, a latency-SLO-aware memory offloading system for LLM serving. A key challenge in designing Select-N is to reconcile the tension between meeting SLOs and maximizing host memory usage. Select-N overcomes it by exploiting a unique characteristic of modern LLMs: during serving, the computation time of each decoder layer is deterministic. Leveraging this, Select-N introduces offloading interval, an internal tunable knob that captures the tradeoff between SLOs and host memory usage, thereby reducing the aforementioned challenge to pick an optimal offloading interval. With that, Select-N proposes a two-stage approach to automatically pick the offloading interval. The first stage is offline that generates the range of optimal offloading interval, while the second stage adjusts offloading interval at the granularity of inference iteration based on runtime hardware status. Our evaluation shows that Select-N consistently meets SLOs and improves the serving throughput over existing mechanisms by 1.85X due to maximizing the use of host memory.