Generative AI in Simulation-Based Test Environments for Large-Scale Cyber-Physical Systems: An Industrial Study

TL;DR

This paper explores how generative AI can enhance simulation-based testing for large-scale cyber-physical systems by capturing industry insights, identifying challenges, and proposing future research directions to foster responsible adoption.

Contribution

It provides detailed industry perspectives on challenges and a research agenda for integrating generative AI into simulation-based testing of cyber-physical systems.

Findings

Practitioners see potential in generative AI for testing.

Identified key challenges in trustworthiness and integration.

Proposed high-priority research directions for future work.

Abstract

Quality assurance for large-scale cyber-physical systems relies on sophisticated test activities using complex test environments investigated with the help of numerous types of simulators. As these systems grow, extensive resources are required to develop and maintain simulation models of hardware and software components, as well as physical environments. Meanwhile, recent advances in generative AI have led to tools that can produce executable test cases for software systems, offering potential benefits such as reducing manual efforts or increasing test coverage. However, the application of generative AI techniques to simulation-based testing of large-scale cyber-physical systems remains underexplored. To better understand this gap, this study captures practitioners' perspectives on leveraging generative AI, based on a cross-company workshop with six organizations. Our contribution is twofold: (1) detailed, experience-based insights into challenges faced by engineers, and (2) a research agenda comprising three high-priority directions: (a) AI-generated scenarios and environment models, (b) simulators and AI in CI/CD pipelines, and (c) trustworthiness in generative AI for simulation. While participants acknowledged substantial potential, they also highlighted unresolved challenges. By detailing these issues, the paper aims to guide future academia-industry collaboration towards the responsible adoption of generative AI in simulation-based testing.