Efficient Multi-Adapter LLM Serving via Cross-Model KV-Cache Reuse with Activated LoRA

TL;DR

This paper introduces a novel LLM serving engine that enables efficient adapter switching by reusing key-value caches across models using Activated LoRA, significantly reducing latency and computation overhead.

Contribution

The work presents the first implementation of cross-model KV-cache reuse with Activated LoRA in a production inference engine, improving efficiency during multi-adapter LLM serving.

Findings

Up to 58x latency reduction in multi-adapter pipelines

Over 100x improvement in time-to-first-token

Benefits scale with model size and sequence length

Abstract

Modern large language model (LLM) systems increasingly rely on multi-turn pipelines that are composed of multiple task-specific adapters, yet existing serving frameworks remain inefficient, incurring substantial recomputation overhead when switching between adapters. We present the first LLM serving engine that supports cross-model prefix cache reuse between base and adapted models via Activated LoRA (aLoRA), enabling efficient and fine-grained adapter switching during inference. Our design extends the vLLM framework by introducing base-aligned block hashing and activation-aware masking within the model execution path, permitting cache reuse across models while preserving compatibility with existing serving engine optimizations. Integrated into a production-grade inference stack, this approach supports dynamic adapter activation without excessive key-value tensor recomputation. Evaluation across representative multi-turn, multi-adapter pipelines demonstrates up to 58x end-to-end latency reduction and over 100x time-to-first-token improvement relative to standard LoRA baselines, with benefits that scale with model size and sequence length and manifest across all stages of the request lifecycle. This work bridges parameter-efficient model adaptation with high-performance serving, providing the first complete realization of cross-model KV-cache reuse in modern LLM inference engines.