Can LLMs Understand Computer Networks? Towards a Virtual System Administrator

TL;DR

This study systematically evaluates Large Language Models' ability to understand and answer questions about computer networks, revealing promising accuracy but highlighting challenges with complex topologies.

Contribution

First comprehensive assessment of LLMs' understanding of computer networks, including a framework and insights into prompt engineering to improve performance.

Findings

Best model achieved 79.3% accuracy in zero-shot scenarios

Proprietary LLMs perform well on small and medium networks

Open-source models struggle with complex network topologies

Abstract

Recent advancements in Artificial Intelligence, and particularly Large Language Models (LLMs), offer promising prospects for aiding system administrators in managing the complexity of modern networks. However, despite this potential, a significant gap exists in the literature regarding the extent to which LLMs can understand computer networks. Without empirical evidence, system administrators might rely on these models without assurance of their efficacy in performing network-related tasks accurately. In this paper, we are the first to conduct an exhaustive study on LLMs' comprehension of computer networks. We formulate several research questions to determine whether LLMs can provide correct answers when supplied with a network topology and questions on it. To assess them, we developed a thorough framework for evaluating LLMs' capabilities in various network-related tasks. We evaluate our framework on multiple computer networks employing proprietary (e.g., GPT4) and open-source (e.g., Llama2) models. Our findings in general purpose LLMs using a zero-shot scenario demonstrate promising results, with the best model achieving an average accuracy of 79.3%. Proprietary LLMs achieve noteworthy results in small and medium networks, while challenges persist in comprehending complex network topologies, particularly for open-source models. Moreover, we provide insight into how prompt engineering can enhance the accuracy of some tasks.