Sutradhara: An Intelligent Orchestrator-Engine Co-design for Tool-based Agentic Inference

TL;DR

Sutradhara is a co-designed system that optimizes tool-based agentic inference by integrating orchestration with LLM serving, reducing latency and increasing throughput in complex AI workloads.

Contribution

It introduces a novel API and three key optimizations that enable cross-layer improvements in tool invocation, caching, and parallelism for agentic LLM applications.

Findings

Up to 77% higher load capacity at same latency

Median FTR latency reduced by up to 15%

End-to-end latency decreased by up to 11% on A100 GPUs

Abstract

Agentic applications are LLMs that iteratively invoke external tools to accomplish complex tasks. Such tool-based agents are rapidly becoming the dominant paradigm for deploying language models in production. Unlike traditional single-turn inference, agentic workloads chain together multiple LLM calls and tool executions before producing a final response, creating a new performance bottleneck that manifests as increased latency in First Token Rendered (FTR) of the final answer. Through analysis of requests at production scale, we reveal three critical challenges: tool calls account for 30-85% of FTR latency, KV cache hit rates collapse despite substantial context reuse across iterations, and sequential orchestration wastes potential intra-request parallelism. These bottlenecks stem from a design gap in which orchestrators and LLM engines operate as decoupled black boxes, preventing cross-layer optimizations. We present Sutradhara, a co-designed agentic inference system that integrates orchestration with LLM serving through a thin API enabling three optimizations: overlap tool execution with subsequent LLM prefill using tool-aware prompt splitting, streaming tool execution to dispatch tools incrementally during decode rather than waiting for complete output, and orchestrator-aware cache management that uses semantic hints to improve hit rates and reduce thrashing. Implemented on vLLM, Sutradhara improves the throughput-latency trade-off in agentic systems, sustains up to 77% higher load at the same median FTR latency, or reduces median FTR latency by up to 15% at the same load while reducing end-to-end latency by upto 11% on A100 GPUs.