TelcoAgent-Bench: A Multilingual Benchmark for Telecom AI Agents

TL;DR

This paper introduces TelcoAgent-Bench, a multilingual benchmarking framework for evaluating telecom-specific LLM agents, focusing on intent recognition, process accuracy, and stability across English and Arabic scenarios.

Contribution

It presents a structured set of metrics and a framework specifically designed to assess the reliability and operational consistency of telecom LLM agents in multilingual environments.

Findings

Recent models understand telecom issues reasonably well.

Models struggle with consistent troubleshooting and stability.

Performance drops in bilingual and unconstrained scenarios.

Abstract

The integration of large language model (LLM) agents into telecom networks introduces new challenges, related to intent recognition, tool execution, and resolution generation, while taking into consideration different operational constraints. In this paper, we introduce TelcoAgent-Bench and TelcoAgent-Metrics, a Telecom-specific benchmarking framework for evaluating multilingual telecom LLM agents. The proposed framework assesses the semantic understanding as well as process-level alignment with structured troubleshooting flows and stability across repeated scenario variations. Our contribution includes a structured suite of metrics that assess intent recognition, ordered tool execution, resolution correctness, and stability across scenario variations, with the aim of quantifying the reliability and operational consistency of LLM agents in telecom environments. The framework is designed to operate in both English and Arabic, to address the need for multilingual agent deployment in operational network environments. Our experimental results show that although recent instruct-tuned models can understand telecom problems in a reasonable way, they usually struggle to consistently follow the required troubleshooting steps and to maintain stable behavior when exposed to different variations of the same scenario. This performance gap becomes more pronounced in unconstrained and bilingual settings.