PrefillShare: A Shared Prefill Module for KV Reuse in Multi-LLM Disaggregated Serving

TL;DR

PrefillShare is a novel method that enables sharing of the prefill stage across multiple models in disaggregated multi-LLM systems, significantly reducing latency and increasing throughput while maintaining accuracy.

Contribution

It introduces a prefill sharing algorithm that factorizes models, freezes the prefill module, and fine-tunes only the decode module, enabling efficient multi-model reuse in disaggregated serving.

Findings

Achieves 4.5x lower p95 latency in multi-model workloads.

Attains 3.9x higher throughput compared to baseline.

Maintains full fine-tuning accuracy across various tasks and models.

Abstract

Multi-agent systems increasingly orchestrate multiple specialized language models to solve complex real-world problems, often invoking them over a shared context. This execution pattern repeatedly processes the same prompt prefix across models. Consequently, each model redundantly executes the prefill stage and maintains its own key-value (KV) cache, increasing aggregate prefill load and worsening tail latency by intensifying prefill-decode interference in existing LLM serving stacks. Disaggregated serving reduces such interference by placing prefill and decode on separate GPUs, but disaggregation does not fundamentally eliminate inter-model redundancy in computation and KV storage for the same prompt. To address this issue, we propose PrefillShare, a novel algorithm that enables sharing the prefill stage across multiple models in a disaggregated setting. PrefillShare factorizes the model into prefill and decode modules, freezes the prefill module, and fine-tunes only the decode module. This design allows multiple task-specific models to share a prefill module and the KV cache generated for the same prompt. We further introduce a routing mechanism that enables effective prefill sharing across heterogeneous models in a vLLM-based disaggregated system. PrefillShare not only matches full fine-tuning accuracy on a broad range of tasks and models, but also delivers 4.5x lower p95 latency and 3.9x higher throughput in multi-model agent workloads.