TeleEval-OS: Performance evaluations of large language models for operations scheduling

TL;DR

This paper introduces TeleEval-OS, a comprehensive benchmark for evaluating large language models in telecommunications operation scheduling, revealing the potential of open-source LLMs to outperform closed-source models in specific tasks.

Contribution

The paper presents the first dedicated benchmark for LLMs in telecom operation scheduling, including diverse datasets and a hierarchical task evaluation framework.

Findings

Open-source LLMs can outperform closed-source LLMs in certain scenarios.

The benchmark covers 15 datasets across 13 subtasks in telecom operations.

LLMs show varying performance across different complexity levels.

Abstract

The rapid advancement of large language models (LLMs) has significantly propelled progress in artificial intelligence, demonstrating substantial application potential across multiple specialized domains. Telecommunications operation scheduling (OS) is a critical aspect of the telecommunications industry, involving the coordinated management of networks, services, risks, and human resources to optimize production scheduling and ensure unified service control. However, the inherent complexity and domain-specific nature of OS tasks, coupled with the absence of comprehensive evaluation benchmarks, have hindered thorough exploration of LLMs' application potential in this critical field. To address this research gap, we propose the first Telecommunications Operation Scheduling Evaluation Benchmark (TeleEval-OS). Specifically, this benchmark comprises 15 datasets across 13 subtasks, comprehensively simulating four key operational stages: intelligent ticket creation, intelligent ticket handling, intelligent ticket closure, and intelligent evaluation. To systematically assess the performance of LLMs on tasks of varying complexity, we categorize their capabilities in telecommunications operation scheduling into four hierarchical levels, arranged in ascending order of difficulty: basic NLP, knowledge Q&A, report generation, and report analysis. On TeleEval-OS, we leverage zero-shot and few-shot evaluation methods to comprehensively assess 10 open-source LLMs (e.g., DeepSeek-V3) and 4 closed-source LLMs (e.g., GPT-4o) across diverse scenarios. Experimental results demonstrate that open-source LLMs can outperform closed-source LLMs in specific scenarios, highlighting their significant potential and value in the field of telecommunications operation scheduling.