CellularSpecSec-Bench: A Staged Benchmark for Evidence-Grounded Interpretation and Security Reasoning over 3GPP Specifications

TL;DR

This paper introduces CellularSpecSec-Bench, a comprehensive benchmark with datasets and a framework for evaluating and advancing security reasoning and interpretation of 3GPP cellular network specifications using language models.

Contribution

It presents a new staged benchmark and a unified framework for systematic understanding and security analysis of cellular specifications, addressing unique challenges in normative language interpretation.

Findings

Developed high-quality, expert-verified datasets for cellular specification analysis.

Established a reproducible benchmark for progress measurement in cellular security reasoning.

Proposed a unified framework for interpretation and security reasoning over 3GPP specifications.

Abstract

Cellular networks are critical infrastructure supporting billions of worldwide users and safety- and mission-critical services. Vulnerabilities in cellular networks can therefore cause service disruption, privacy breaches, and broad societal harm, motivating growing efforts to analyze 3GPP specifications that define required device and operator behavior. While large language models (LLMs) have demonstrated the capability for reading technical documents, cellular specifications impose unique challenges: faithful interpretation of normative language, reasoning across cross-referenced clauses, and verifiable conclusions grounded in multimodal evidence such as tables and figures. To address these challenges, we propose CellSpecSec-ARI, a unified Adapt-Retrieve-Integrate framework for systematic understanding and standard-driven security analysis of 3GPP specifications; CellularSpecSec-Bench, a staged benchmark, containing newly constructed high-quality datasets with expert-verified and corrected subsets from prior open-source resources. Together, they establish an accessible and reproducible foundation for quantifying progress in specification understanding and security reasoning in the cellular network security domain.