Can LLMs Understand Computer Networks? Towards a Virtual System Administrator
Denis Donadel, Francesco Marchiori, Luca Pajola, Mauro Conti

TL;DR
This study systematically evaluates Large Language Models' ability to understand and answer questions about computer networks, revealing promising accuracy but highlighting challenges with complex topologies.
Contribution
First comprehensive assessment of LLMs' understanding of computer networks, including a framework and insights into prompt engineering to improve performance.
Findings
Best model achieved 79.3% accuracy in zero-shot scenarios
Proprietary LLMs perform well on small and medium networks
Open-source models struggle with complex network topologies
Abstract
Recent advancements in Artificial Intelligence, and particularly Large Language Models (LLMs), offer promising prospects for aiding system administrators in managing the complexity of modern networks. However, despite this potential, a significant gap exists in the literature regarding the extent to which LLMs can understand computer networks. Without empirical evidence, system administrators might rely on these models without assurance of their efficacy in performing network-related tasks accurately. In this paper, we are the first to conduct an exhaustive study on LLMs' comprehension of computer networks. We formulate several research questions to determine whether LLMs can provide correct answers when supplied with a network topology and questions on it. To assess them, we developed a thorough framework for evaluating LLMs' capabilities in various network-related tasks. We evaluate…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsDigital Rights Management and Security
