ExpertWeave: Efficiently Serving Expert-Specialized Fine-Tuned Adapters at Scale

TL;DR

ExpertWeave is a system that enables efficient, scalable serving of multiple expert-specialized adapters for large language models, significantly reducing memory usage and increasing throughput with minimal latency overhead.

Contribution

It introduces a novel system that allows concurrent serving of multiple ESFT adapters over a shared MoE base model with minimal resource overhead and seamless integration.

Findings

Can serve multiple adapters on a single accelerator where baseline fails

Achieves up to 94x more KV cache capacity and 18% higher throughput

Maintains low latency overhead even with 20 adapters

Abstract

Expert-Specialized Fine-Tuning (ESFT) adapts Mixture-of-Experts (MoE) large language models to enhance their task-specific performance by selectively tuning the top-activated experts for the task. Serving these fine-tuned models at scale is challenging: deploying merged models in isolation is prohibitively resource-hungry, while existing multi-adapter serving systems with LoRA-style additive updates are incompatible with ESFT's expert-oriented paradigm. We present ExpertWeave, a system that serves multiple ESFT adapters concurrently over a single shared MoE base model, drastically reducing the memory footprint and improving resource utilization. To seamlessly integrate into existing inference pipelines for MoE models with non-intrusive modifications and minimal latency overhead, ExpertWeave introduces a virtual-memory-assisted expert weight manager that co-locates base-model and adapter experts without incurring memory overhead from fragmentation, and a fused kernel for batched rerouting to enable lightweight redirection of tokens to the appropriate experts at runtime. Our evaluations show that ExpertWeave can simultaneously serve multiple adapters of a 16B MoE model on a single accelerator where the baseline runs out of memory, or provides up to 94x more KV cache capacity and achieves up to 18% higher throughput while using comparable resources, all without compromising model accuracy. ExpertWeave maintains low overhead even when scaling to 20 adapters, with a 4-11% latency increase compared with serving the base model alone. Source code will be released soon.