Serving Heterogeneous LoRA Adapters in Distributed LLM Inference Systems

TL;DR

LoRAServe is a dynamic, workload-aware system that efficiently manages heterogeneous LoRA adapters in distributed LLM inference, significantly improving throughput and latency while reducing GPU usage.

Contribution

It introduces LoRAServe, a novel framework for dynamic adapter placement and routing that addresses rank heterogeneity in LoRA serving systems.

Findings

Up to 2× higher throughput achieved.

Up to 9× lower tail latency (TTFT).

Uses up to 50% fewer GPUs under SLO constraints.

Abstract

Low-Rank Adaptation (LoRA) has become the de facto method for parameter-efficient fine-tuning of large language models (LLMs), enabling rapid adaptation to diverse domains. In production, LoRA-based models are served at scale, creating multi-tenant environments with hundreds of adapters sharing a base model. However, state-of-the-art serving systems co-batch heterogeneous adapters without accounting for rank (size) variability, leading to severe performance skew, which ultimately requires adding more GPUs to satisfy service-level objectives (SLOs). Existing optimizations, focused on loading, caching, and kernel execution, ignore this heterogeneity, leaving GPU resources underutilized. We present LoRAServe, a workload-aware dynamic adapter placement and routing framework designed to tame rank diversity in LoRA serving. By dynamically rebalancing adapters across GPUs and leveraging GPU Direct RDMA for remote access, LoRAServe maximizes throughput and minimizes tail latency under real-world workload drift. Evaluations on production traces from Company X show that LoRAServe elicits up to 2$\times$ higher throughput, up to 9$\times$ lower TTFT, while using up to 50% fewer GPUs under SLO constraints compared to state-of-the-art systems.