NetAgentBench: A State-Centric Benchmark for Evaluating Agentic Network Configuration

TL;DR

NetAgentBench is a dynamic, FSM-based benchmark designed to evaluate multi-turn agent interactions in network management, revealing critical deficiencies in current LLM agents for complex tasks.

Contribution

Introduces a formal, deterministic benchmark framework for assessing agentic network management, emphasizing multi-turn behavioral stability and robustness.

Findings

Empirical evaluation shows current LLM agents struggle with complex network tasks.

Agents perform adequately on basic tasks but fail at expert-level configurations.

Systematic evaluation exposes exploration and coherence issues in agent behaviors.

Abstract

As agentic network management gains popularity, there is a critical need for evaluation frameworks that transcend static, one-shot testing. To address this, we introduce NetAgentBench, a dynamic benchmark that evaluates agent interactions through a Finite State Machine (FSM) formalization guaranteeing determinism, correctness, and bounded execution. This provides the networking landscape with a rigorous foundation to measure complex, multi-turn operational behaviors. Our empirical evaluation of four state-of-the-art LLM agents through diverse network configuration tasks reveals stark deficiencies: while agents can solve basic tasks, they suffer severe exploration meltdowns and coherence collapse during expert-level configurations. Ultimately, NetAgentBench demonstrates that systematically evaluating multi-turn behavioral stability is an indispensable step toward realizing trustworthy, fully autonomous networks.