SKILLS: Structured Knowledge Injection for LLM-Driven Telecommunications Operations

TL;DR

This paper presents SKILLS, a benchmark framework for evaluating LLMs in telecom workflows, demonstrating that structured domain guidance significantly improves model performance across diverse scenarios.

Contribution

Introduction of SKILLS, a comprehensive benchmark with real API simulations and evaluation metrics, to assess LLM capabilities in telecom operations with structured domain knowledge.

Findings

Structured guidance improves LLM performance by up to 18.9 percentage points.

MiniMax M2.5 achieves 81.1% success rate with domain skills.

Models show consistent skill lift across multiple scenarios.

Abstract

As telecommunications operators accelerate adoption of AI-enabled automation, a practical question remains unresolved: can general-purpose large language model (LLM) agents reliably execute telecom operations workflows through real API interfaces, or do they require structured domain guidance? We introduce SKILLS (Structured Knowledge Injection for LLM-driven Service Lifecycle operations), a benchmark framework comprising 37 telecom operations scenarios spanning 8 TM Forum Open API domains (TMF620, TMF621, TMF622, TMF628, TMF629, TMF637, TMF639, TMF724). Each scenario is grounded in live mock API servers with seeded production-representative data, MCP tool interfaces, and deterministic evaluation rubrics combining response content checks, tool-call verification, and database state assertions. We evaluate open-weight models under two conditions: baseline (generic agent with tool access but no domain guidance) and with-skill (agent augmented with a portable SKILL.md document encoding workflow logic, API patterns, and business rules). Results across 5 open-weight model conditions and 185 scenario-runs show consistent skill lift across all models. MiniMax M2.5 leads (81.1% with-skill, +13.5pp), followed by Nemotron 120B (78.4%, +18.9pp), GLM-5 Turbo (78.4%, +5.4pp), and Seed 2.0 Lite (75.7%, +18.9pp).