AutonomyLens: A Self-Evolving Simulation-Based Testing Loop for Autonomous Systems

TL;DR

AutonomyLens is an LLM-driven framework that unifies scenario design, simulation, and telemetry analysis to improve validation, traceability, and reproducibility of autonomous systems.

Contribution

It introduces a structured scenario representation, automated execution, context-aware analysis, and counterfactual scenario generation for autonomous system validation.

Findings

Framework enables high-level validation translation into executable scenarios.

Automated pipeline improves simulation efficiency and consistency.

Counterfactual generation refines testing from observed failures.

Abstract

Software engineering practices for validating autonomous cyber-physical systems (e.g., Uncrewed Aerial Vehicles) remain fragmented across scenario design, simulation execution, and telemetry analysis, limiting traceability between requirements, tests, and evidence. This fragmentation reduces reproducibility, slows debugging and iteration, and hinders systematic assurance under complex and evolving environmental conditions. We present AutonomyLens, an LLM-driven framework that integrates scenario specification, simulation execution, and telemetry analysis into a unified validation workflow. AutonomyLens enables developers to translate high-level validation intent into executable, temporally evolving scenarios, automatically run simulations, and perform context-aware analysis of resulting system behavior. The framework introduces (i) a structured representation for mission-level scenarios, (ii) an automated execution pipeline, (iii) analysis mechanisms that align telemetry with scenario context to produce actionable insights, and (iv) counterfactual scenario generation that closes the loop by refining and synthesizing new test cases from observed failures. We describe the early-stage design of AutonomyLens, discuss key challenges in building integrated validation workflows for autonomous systems, and outline how such an approach can improve traceability, reproducibility, and scalability in autonomy validation.