OOCO: Latency-disaggregated Architecture for Online-Offline Co-locate LLM Serving

TL;DR

This paper introduces OOCO, a latency-aware architecture for co-locating online and offline LLM workloads, which improves resource utilization and throughput while maintaining strict online latency SLOs.

Contribution

It proposes a novel latency-disaggregated architecture with a specialized scheduler and preemption mechanism to balance online and offline LLM serving workloads.

Findings

Offline throughput increased by up to 3x.

Online request SLOs are strictly maintained.

Effective handling of bursty traffic patterns.

Abstract

Large Language Models (LLMs) are increasingly deployed in both latency-sensitive online services and cost-sensitive offline workloads. Co-locating these workloads on shared serving instances can improve resource utilization, but directly applying this approach to Prefill/Decode (P/D) disaggregated systems introduces severe load imbalance, as fluctuating request mixes alter the intrinsic P/D ratio. Existing dynamic adjustment techniques cannot keep up with the bursty traffic patterns of online services. We propose a latency-constraint disaggregated architecture, which separates cluster resources into latency-strict and latency-relaxed pools based on task latency requirements. This design enables flexible placement of offline decode tasks, mitigating P/D imbalance while preserving online performance. To fully exploit this flexibility, we propose (1) a bottleneck-based scheduler guided by a Roofline-based performance model for performance bottleneck based scheduling, and (2) a fast preemption mechanism that strictly enforces Service Level Objectives (SLOs) for online requests. Experiments on real-world traces show that compared to existing offline system approaches, our method improves offline throughput by up to 3x, while maintaining online request SLOs.