Trinity: Disaggregating Vector Search from Prefill-Decode Disaggregation in LLM Serving

TL;DR

Trinity is a framework that optimizes large language model serving by integrating vector search with prefill-decode disaggregation, reducing latency and improving GPU utilization.

Contribution

It introduces a novel architecture for GPU-based vector search within PD disaggregated LLM serving, including continuous batching and stage-aware scheduling.

Findings

Unified vector search GPU pool reduces tail latency.

Continuous batching improves GPU utilization under heterogeneous workloads.

Stage-aware scheduling effectively preempts vector search requests.

Abstract

Prefill and decode (PD) disaggregation separates prompt prefill and token-by-token decode stages into distinct GPU pools and has become the dominant architecture for large-scale LLM serving in industry. Also, retrieval tasks via vector search remains entangled with the model inference process, like heterogeneous RAG requests and prompt answer caches, inflating tail latency. We are motivated to investigate how vector search should be orchestrated along with PD disaggregation with a dedicated deployment architecture without violating SLOs in various retrieval workloads. We present Trinity, a practical framework that consolidates all retrieval into a single, shared vector-search GPU pool and make it work with PD disaggregated LLM serving in match. Trinity introduces (1) a novel architecture for deploying GPU-based vector search service in PD disaggregation. (2) Continuous batching for vector search that make full used of GPUs under heterogeneous queries; (3) Stage-aware scheduling that preempts vector search requests between both decode and prefill tasks.