TeleTables: A Benchmark for Large Language Models in Telecom Table Interpretation

TL;DR

TeleTables is a new benchmark for evaluating large language models' ability to interpret and reason over telecom tables in technical standards, revealing limitations of current models and emphasizing the need for domain-specific fine-tuning.

Contribution

We introduce TeleTables, a comprehensive benchmark dataset for assessing LLMs' knowledge and interpretation of telecom tables, created through a novel multi-stage data generation pipeline.

Findings

Smaller models under 10B parameters struggle with telecom knowledge and table interpretation.

Larger models demonstrate better reasoning capabilities on telecom tables.

Domain-specific fine-tuning is essential for reliable telecom standards interpretation.

Abstract

Language Models (LLMs) are increasingly explored in the telecom industry to support engineering tasks, accelerate troubleshooting, and assist in interpreting complex technical documents. However, recent studies show that LLMs perform poorly on telecom standards, particularly 3GPP specifications. We argue that a key reason is that these standards densely include tables to present essential information, yet the LLM knowledge and interpretation ability of such tables remains largely unexamined. To address this gap, we introduce TeleTables, a benchmark designed to evaluate both the implicit knowledge LLMs have about tables in technical specifications and their explicit ability to interpret them. TeleTables is built through a novel multi-stage data generation pipeline that extracts tables from 3GPP standards and uses multimodal and reasoning-oriented LLMs to generate and validate questions. The resulting dataset, which is publicly available, comprises 500 human-verified question-answer pairs, each associated with the corresponding table in multiple formats. Our evaluation shows that, smaller models (under 10B parameters) struggle both to recall 3GPP knowledge and to interpret tables, indicating the limited exposure to telecom standards in their pretraining and the insufficient inductive biases for navigating complex technical material. Larger models, on the other hand, show stronger reasoning on table interpretation. Overall, TeleTables highlights the need for domain-specialized fine-tuning to reliably interpret and reason over telecom standards.