GhostServe: A Lightweight Checkpointing System in the Shadow for Fault-Tolerant LLM Serving

TL;DR

GhostServe is a lightweight checkpointing system that enhances fault tolerance in large language model serving by efficiently protecting and recovering the KV cache using erasure coding, reducing latency and improving availability.

Contribution

It introduces GhostServe, a novel erasure coding-based checkpointing approach for fault-tolerant LLM inference, addressing the vulnerability of the KV cache in distributed systems.

Findings

Reduces checkpointing latency by up to 2.7x

Decreases recovery latency by 2.1x for a single batch

Achieves 1.2x median response latency improvement during failures

Abstract

The rise of million-token, agent-based applications has placed unprecedented demands on large language model (LLM) inference services. The long-running nature of these tasks increases their susceptibility to hardware and software faults, leading to costly job failures, wasted resources, and degraded user experience. The stateful key-value (KV) cache, which grows with the sequence length, presents a central challenge as it is a critical and vulnerable component in distributed serving systems. In this work, we propose GhostServe, a novel checkpointing solution to facilitate fault-tolerant LLM serving. Specifically, GhostServe protects the streaming KV cache in the shadow by applying erasure coding to generate and store the parity shards in host memory. In the event of device failures, GhostServe enables fast reconstruction of the lost KV cache, allowing the inference process to resume seamlessly without costly full recomputation or state replication. Evaluations demonstrate that GhostServe reduces checkpointing latency by up to 2.7x and recovery latency by 2.1x for a single batch, and 1.2x median response latency compared to existing methods, in the presence of system failures, paving the way for high-availability and cost-effective LLM serving at scale.