Nalar: An agent serving framework

TL;DR

Nalar is a comprehensive agent-serving framework that enhances efficiency, scalability, and robustness for complex, multi-step agentic applications by separating workflow specification from execution and providing advanced runtime control.

Contribution

It introduces a novel framework that decouples workflow design from execution, enabling scalable, efficient, and policy-driven serving of heterogeneous agent applications.

Findings

Reduces tail latency by 34-74% across workloads.

Achieves up to 2.9x speedups over baselines.

Supports scaling to 130K futures with low control overhead.

Abstract

LLM-driven agentic applications increasingly automate complex, multi-step tasks, but serving them efficiently remains challenging due to heterogeneous components, dynamic and model-driven control flow, long-running state, and unpredictable latencies. Nalar is a ground-up agent-serving framework that cleanly separates workflow specification from execution while providing the runtime visibility and control needed for robust performance. Nalar preserves full Python expressiveness, using lightweight auto-generated stubs that turn agent and tool invocations into futures carrying dependency and context metadata. A managed state layer decouples logical state from physical placement, enabling safe reuse, migration, and consistent retry behavior. A two-level control architecture combines global policy computation with local event-driven enforcement to support adaptive routing, scheduling, and resource management across evolving workflows. Together, these mechanisms allow Nalar to deliver scalable, efficient, and policy-driven serving of heterogeneous agentic applications without burdening developers with orchestration logic. Across three agentic workloads, Nalar cuts tail latency by 34--74\%, achieves up to $2.9\times$ speedups, sustains 80 RPS where baselines fail, and scales to 130K futures with sub-500 ms control overhead.