A Search-Based Framework for Automatic Generation of Testing Environments for Cyber-Physical Systems

TL;DR

This paper presents AmbieGen, a search-based framework that automatically generates diverse and fault-revealing test environments for autonomous cyber-physical systems using multi-objective genetic algorithms.

Contribution

The paper introduces AmbieGen, a novel search-based framework employing multi-objective genetic algorithms to generate diverse test scenarios for autonomous cyber-physical systems.

Findings

Multi-objective configuration produces more diverse test scenarios.

AmbieGen outperforms random search in scenario quality.

Multi-objective approach matches single-objective quality with greater diversity.

Abstract

Many modern cyber physical systems incorporate computer vision technologies, complex sensors and advanced control software, allowing them to interact with the environment autonomously. Testing such systems poses numerous challenges: not only should the system inputs be varied, but also the surrounding environment should be accounted for. A number of tools have been developed to test the system model for the possible inputs falsifying its requirements. However, they are not directly applicable to autonomous cyber physical systems, as the inputs to their models are generated while operating in a virtual environment. In this paper, we aim to design a search based framework, named AmbieGen, for generating diverse fault revealing test scenarios for autonomous cyber physical systems. The scenarios represent an environment in which an autonomous agent operates. The framework should be applicable to generating different types of environments. To generate the test scenarios, we leverage the NSGA II algorithm with two objectives. The first objective evaluates the deviation of the observed system behaviour from its expected behaviour. The second objective is the test case diversity, calculated as a Jaccard distance with a reference test case. We evaluate AmbieGen on three scenario generation case studies, namely a smart-thermostat, a robot obstacle avoidance system, and a vehicle lane keeping assist system. We compared three configurations of AmbieGen: based on a single objective genetic algorithm, multi objective, and random search. Both single and multi objective configurations outperform the random search. Multi objective configuration can find the individuals of the same quality as the single objective, producing more unique test scenarios in the same time budget.