RANalyzer: Automated Continuous RAN Software Evaluation and Regression Analysis

TL;DR

RANalyzer is an automated framework that uses semantic analysis and residuals to evaluate the performance impact of software updates in O-RAN systems, enabling scalable, continuous testing.

Contribution

It introduces an LLM-assisted semantic extraction combined with residuals analysis to attribute performance variations to specific software changes in wireless systems.

Findings

Analyzed over 8,600 tests across 69 releases over two years.

Successfully correlated performance deviations with specific code change categories.

Framework can be integrated into CI/CD pipelines for automated evaluation.

Abstract

Software-driven O-RAN architectures enable rapid innovation through frequent, independent updates to virtualized components. However, attributing performance variations to specific software changes is challenging due to the stochastic nature of wireless systems, where channel conditions, interference, and hardware variability confound analysis. Traditional threshold-based monitoring and manual troubleshooting do not scale with modern software evolution. This paper presents RANalyzer, an automated test analysis framework that quantifies the performance impact of software updates beyond what can be explained by wireless channel conditions. RANalyzer combines LLM-assisted semantic extraction with residuals analysis. The first categorizes code changes by affected protocol layers and functional components, while the second provides insights on the effect of load, channel, or code changes on the test performance. We contribute an extensive dataset collected over more than two years of continuous over-the-air testing on an experimental O-RAN testbed, comprising over 8,600 automated tests across 69 releases of the OAI stack. By modeling expected performance and interpreting deviations as software-induced effects, we identify degraded instances attributable to code changes and correlate them with specific change categories. The framework can be integrated into CI/CD/CT pipelines for automated, continuous evaluation of software updates at scale.