# Predictable LLM Serving on GPU Clusters

## TL;DR

This paper introduces a host-level controller for GPU clusters that dynamically manages MIG configurations and placement to reduce latency tail and SLO violations for LLM inference, achieving significant improvements over static methods.

## Contribution

It presents a novel, fabric-agnostic, VM-deployable controller that dynamically optimizes GPU resource allocation and placement to improve LLM serving performance and reliability.

## Key findings

- SLO miss-rate reduced by approximately 32%.
- p99 latency improved by approximately 15%.
- 10-15% p99 latency improvement on LLM serving with minimal throughput cost.

## Abstract

Latency-sensitive inference on shared A100 clusters often suffers noisy-neighbor interference on the PCIe fabric, inflating tail latency and SLO violations. We present a fabric-agnostic, VM-deployable host-level controller that combines dynamic Multi-Instance GPU (MIG) reconfiguration, PCIe-aware placement, and lightweight guardrails (MPS quotas, cgroup I/O). It samples per-tenant tails and system signals, uses topology hints to avoid PCIe hot spots, and gates actions with dwell/cool-down to avoid thrash. On a single host and a 2-node (16-GPU) cluster, SLO miss-rate is reduced by \(\approx\)32\% (\(\approx\)1.5) and p99 latency improves \(\approx\)15\% with \(\leq\)5\% throughput cost versus static MIG and naive placement; ablations show MIG and placement contribute comparably. We also evaluate LLM serving with vLLM on OLMo 2 7B Instruct: TTFT p99 improves \(\approx\)10--15\% at \(\leq\)5\% cost without changing the controller.