An Empirical Study of NetOps Capability of Pre-Trained Large Language Models

TL;DR

This paper introduces NetEval, a comprehensive multilingual benchmark for assessing the NetOps capabilities of 26 LLMs, revealing GPT-4's near-human performance and highlighting potential in open models like LLaMA 2.

Contribution

The paper presents NetEval, a new evaluation set specifically designed to measure NetOps capabilities of LLMs across multiple languages and sub-domains.

Findings

GPT-4 achieves near-human performance on NetEval.

Open models like LLaMA 2 show significant potential.

Most LLMs lag behind GPT-4 in NetOps tasks.

Abstract

Nowadays, the versatile capabilities of Pre-trained Large Language Models (LLMs) have attracted much attention from the industry. However, some vertical domains are more interested in the in-domain capabilities of LLMs. For the Networks domain, we present NetEval, an evaluation set for measuring the comprehensive capabilities of LLMs in Network Operations (NetOps). NetEval is designed for evaluating the commonsense knowledge and inference ability in NetOps in a multi-lingual context. NetEval consists of 5,732 questions about NetOps, covering five different sub-domains of NetOps. With NetEval, we systematically evaluate the NetOps capability of 26 publicly available LLMs. The results show that only GPT-4 can achieve a performance competitive to humans. However, some open models like LLaMA 2 demonstrate significant potential.