Data Driven Testing of Cyber Physical Systems

TL;DR

This paper presents a data-driven approach to automatically generate fault-revealing test cases for cyber-physical systems, demonstrated on a smart thermostat to improve testing effectiveness under changing conditions.

Contribution

It introduces a novel method for automatic test case generation for CPS using real-world data and model learning, enhancing fault detection beyond traditional domain knowledge methods.

Findings

Successfully generated fault scenarios for a smart thermostat

Identified several environment conditions causing system failures

Improved testing coverage using data-driven models

Abstract

Consumer grade cyber-physical systems (CPS) are becoming an integral part of our life, automatizing and simplifying everyday tasks. Indeed, due to complex interactions between hardware, networking and software, developing and testing such systems is known to be a challenging task. Various quality assurance and testing strategies have been proposed. The most common approach for pre-deployment testing is to model the system and run simulations with models or software in the loop. In practice, most often, tests are run for a small number of simulations, which are selected based on the engineers' domain knowledge and experience. In this paper we propose an approach to automatically generate fault-revealing test cases for CPS. We have implemented our approach in Python, using standard frameworks and used it to generate scenarios violating temperature constraints for a smart thermostat implemented as a part of our IoT testbed. Data collected from an application managing a smart building have been used to learn models of the environment under ever changing conditions. The suggested approach allowed us to identify several pit-fails, scenarios (i.e., environment conditions and inputs), where the system behaves not as expected.