How Helpful is LLM Assistance in Network Operations? A Case Study at a Large Demonstration Network
Ryo Nakamura, Koshi Eguchi

TL;DR
This case study evaluates how a Large Language Model (LLM) assists over 100 network engineers in building and operating a complex demonstration network, providing insights into its helpfulness and capabilities.
Contribution
The paper presents an extensive real-world experiment assessing LLM assistance in network operations, including user interactions and performance analysis.
Findings
68.1% of chatbot responses were positively evaluated by engineers.
Understanding chatbot capabilities improves response quality.
Detailed use case analyses illustrate practical applications.
Abstract
This paper reports on a real-world case study in which over 100 network engineers assessed how a Large Language Model (LLM) can assist in building and operating a network. The versatility of LLMs has accelerated their adoption across a wide range of domains, and assisting network operations is one such promising application. LLMs are probabilistic models, unlike deterministic protocols and configurations; therefore, clarifying their capabilities -- how and to what extent LLMs can help in network operations -- is a crucial step toward adopting LLMs. To offer practical insights into this issue, we conducted an extensive experiment on a large demonstration network built for a public exhibition, consisting of 21 racks with heterogeneous network devices. In the experiment, a total of 105 network engineers used an LLM-based chatbot while building and operating the network. The chatbot was…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
