LLMs-Powered Real-Time Fault Injection: An Approach Toward Intelligent Fault Test Cases Generation

TL;DR

This paper introduces a novel LLMs-assisted method for generating fault test cases to improve real-time fault injection testing in automotive software, reducing manual effort and enhancing safety verification.

Contribution

The paper proposes a new approach using LLMs, especially GPT-4o, for automatic fault test case generation from safety requirements, improving efficiency and coverage in automotive safety testing.

Findings

High F1-score of 88% for FSR classification

Fault TCs generation accuracy of 97.5%

Effective real-time fault injection on hardware-in-the-loop system

Abstract

A well-known testing method for the safety evaluation and real-time validation of automotive software systems (ASSs) is Fault Injection (FI). In accordance with the ISO 26262 standard, the faults are introduced artificially for the purpose of analyzing the safety properties and verifying the safety mechanisms during the development phase. However, the current FI method and tools have a significant limitation in that they require manual identification of FI attributes, including fault type, location and time. The more complex the system, the more expensive, time-consuming and labour-intensive the process. To address the aforementioned challenge, a novel Large Language Models (LLMs)-assisted fault test cases (TCs) generation approach for utilization during real-time FI tests is proposed in this paper. To this end, considering the representativeness and coverage criteria, the applicability of various LLMs to create fault TCs from the functional safety requirements (FSRs) has been investigated. Through the validation results of LLMs, the superiority of the proposed approach utilizing gpt-4o in comparison to other state-of-the-art models has been demonstrated. Specifically, the proposed approach exhibits high performance in terms of FSRs classification and fault TCs generation with F1-score of 88% and 97.5%, respectively. To illustrate the proposed approach, the generated fault TCs were executed in real time on a hardware-in-the-loop system, where a high-fidelity automotive system model served as a case study. This novel approach offers a means of optimizing the real-time testing process, thereby reducing costs while simultaneously enhancing the safety properties of complex safety-critical ASSs.