LLM-Generated Fault Scenarios for Evaluating Perception-Driven Lane Following in Autonomous Edge Systems

TL;DR

This paper presents a novel offline-online fault injection framework using LLMs and LDMs to evaluate perception-driven lane following in autonomous edge systems, addressing resource constraints and environmental hazards.

Contribution

It introduces a decoupled framework that generates complex fault scenarios offline with LLMs and LDMs, enabling real-time fault-aware inference on edge devices.

Findings

Fault scenarios reveal significant robustness degradation in lane-following models.

Model accuracy drops by up to 99% under fault conditions.

Normal data evaluation is insufficient for real-world robustness assessment.

Abstract

Deploying autonomous vision systems on edge devices faces a critical challenge: resource constraints prevent real-time and predictable execution of comprehensive safety tests. Existing validation methods depend on static datasets or manual fault injection, failing to capture the diverse environmental hazards encountered in real-world deployment. To address this, we introduce a decoupled offline-online fault injection framework. This architecture separates the validation process into two distinct phases: a computationally intensive Offline Phase and a lightweight Online Phase. In the offline phase, we employ Large Language Models (LLMs) to semantically generate structured fault scenarios and Latent Diffusion Models (LDMs) to synthesize high-fidelity sensor degradations. These complex fault dynamics are distilled into a pre-computed lookup table, enabling the edge device to perform real-time fault-aware inference without running heavy AI models locally. We extensively validated this framework on a ResNet18 lane-following model across 460 fault scenarios. Results show that while the model achieves a baseline R^2 of approximately 0.85 on clean data, our generated faults expose significant robustness degradation, with RMSE increasing by up to 99% and within-0.10 localization accuracy dropping to as low as 31.0% under fog conditions, demonstrating the inadequacy of normal-data evaluation for real-world edge AI deployment.