BlendServe: Optimizing Offline Inference for Auto-regressive Large Models with Resource-aware Batching

TL;DR

BlendServe is a system that enhances offline inference throughput for large auto-regressive models by intelligently combining resource overlapping and prefix sharing, leading to significant performance improvements.

Contribution

It introduces a resource-aware prefix tree to optimize request scheduling, balancing resource utilization and prefix sharing in offline batch inference.

Findings

Achieves up to 1.44x throughput increase over industry standards

Effectively balances resource overlapping and prefix sharing

Demonstrates benefits on synthetic multi-modal workloads

Abstract

Offline batch inference, which leverages the flexibility of request batching to achieve higher throughput and lower costs, is becoming more popular for latency-insensitive applications. Meanwhile, recent progress in model capability and modality makes requests more diverse in compute and memory demands, creating unique opportunities for throughput improvement by resource overlapping. However, a request schedule that maximizes resource overlapping can conflict with the schedule that maximizes prefix sharing, a widely-used performance optimization, causing sub-optimal inference throughput. We present BlendServe, a system that maximizes resource utilization of offline batch inference by combining the benefits of resource overlapping and prefix sharing using a resource-aware prefix tree. BlendServe exploits the relaxed latency requirements in offline batch inference to reorder and overlap requests with varied resource demands while ensuring high prefix sharing. We evaluate BlendServe on a variety of synthetic multi-modal workloads and show that it provides up to $1.44\times$ throughput boost compared to widely-used industry standards, vLLM and SGLang.