HFX: Joint Design of Algorithms and Systems for Multi-SLO Serving and Fast Scaling

TL;DR

HFX is a system for large language model serving that jointly optimizes request scheduling and elastic scaling to meet diverse user-specific SLOs efficiently.

Contribution

HFX introduces a proactive scheduler and fast D2D weight transfer scaler, enabling multi-SLO serving with improved latency, SLO compliance, and cost savings in real-world deployments.

Findings

HFX achieves up to 4.44× higher SLO attainment.

Reduces end-to-end latency by up to 65.82%.

Lowers NPU usage cost by up to 49.81%.

Abstract

Large language model (LLM) serving faces the dual challenge of meeting strict user-specific service-level objectives (SLOs) while minimizing computational cost under dynamic, multi-task workloads. Existing approaches either rely on static scheduling policies or focus on single-task settings, limiting their applicability in real-world deployments with heterogeneous requests, variable prompt lengths, and elastic scaling requirements. We present HFX, a production LLM serving system that jointly optimizes request scheduling and elastic scaling across model replicas to satisfy diverse SLOs. HFX introduces a \textbf{scheduler} that performs proactive budget estimation and prioritization to ensure SLO compliance for both new and in-flight requests. HFX also integrates a \textbf{scaler} that supports fast device-to-device (D2D) weight transfer, reducing cold-start latency. Additionally, the system supports both colocated and disaggregated prefill/decode deployments, enabling adaptation to diverse workload patterns and cloud environments. Through extensive experiments on multi-task workloads, we demonstrate consistently higher SLO attainment, lower end-to-end latency, and lower NPU usage cost by up to 4.44$\times$, 65.82\%, and 49.81\%, respectively, compared to state-of-the-art systems. Our results highlight the effectiveness of SLO-aware scheduling and scaling in practical LLM serving, providing a robust framework for cost-efficient and SLO-compliant deployments.