Test Case Generation for Simulink Models: An Experience from the E-Bike Domain

TL;DR

This paper evaluates the effectiveness of the HECATE search-based testing framework in generating failure-revealing test cases for Simulink models of e-Bike controllers, providing empirical evidence from industrial domain applications.

Contribution

It offers an empirical assessment of HECATE's performance on real-world e-Bike models, demonstrating its practical utility in industry-specific cyber-physical systems testing.

Findings

HECATE identified failures in 83% of experiments.

Average time to generate test cases was approximately 1 hour 17 minutes.

Failures identified by HECATE were confirmed by developers.

Abstract

Cyber-physical systems development often requires engineers to search for defects in their Simulink models. Search-based software testing (SBST) is a standard technology that supports this activity. To increase practical adaption, industries need empirical evidence of the effectiveness and efficiency of (existing) SBST techniques on benchmarks from different domains and of varying complexity. To address this industrial need, this paper presents our experience assessing the effectiveness and efficiency of SBST in generating failure-revealing test cases for cyber-physical systems requirements. Our study subject is within the electric bike (e-Bike) domain and concerns the software controller of an e-Bike motor, particularly its functional, regulatory, and safety requirements. We assessed the effectiveness and efficiency of HECATE, an SBST framework for Simulink models, to analyze two software controllers. HECATE successfully identified failure-revealing test cases for 83% (30 out of 36) of our experiments. It required, on average, 1 h 17 min 26 s (min = 11 min 56 s, max = 8 h 16 min 22 s, std = 1 h 50 min 34 s) to compute the failure-revealing test cases. The developer of the e-Bike model confirmed the failures identified by HECATE. We present the lessons learned and discuss the relevance of our results for industrial applications, the state of practice improvement, and the results' generalizability.