SAIR: Cost-Efficient Multi-Stage ML Pipeline Autoscaling via In-Context Reinforcement Learning

TL;DR

SAIR is a novel autoscaling framework for multi-stage ML inference pipelines that leverages in-context reinforcement learning with an LLM to optimize resource allocation and latency without offline training.

Contribution

SAIR introduces an in-context RL controller using LLMs for online policy improvement in autoscaling, combining reward shaping, surprisal-guided retrieval, and fine-grained GPU control.

Findings

Achieves up to 50% P99 latency reduction

Reduces effective resource cost by up to 97%

Detects bottlenecks with 86% accuracy

Abstract

Multi-stage ML inference pipelines are difficult to autoscale due to heterogeneous resources, cross-stage coupling, and dynamic bottleneck migration. We present SAIR, an autoscaling framework that uses an LLM as an in-context reinforcement learning controller, improving its policy online from reward-labeled interaction histories without gradient updates. SAIR combines Pareto-dominance reward shaping with a provable separation margin, surprisal-guided experience retrieval for context efficiency, and fine-grained GPU rate control via user-space CUDA interception. We provide regret analysis decomposing error into retrieval coverage and LLM selection components. On four ML serving pipelines under three workload patterns, SAIR achieves the best or tied-best P99 latency and effective resource cost among deployed baselines, improving P99 by up to 50% and reducing effective cost by up to 97% (under GPU rate-control assumptions), with 86% bottleneck detection accuracy and no offline training.