Exploring the Potential of Large Language Models in Simulink-Stateflow Mutant Generation

TL;DR

This paper explores using Large Language Models to generate high-quality mutants for Simulink-Stateflow models, addressing limitations of traditional mutation techniques in safety-critical Cyber-Physical Systems.

Contribution

It introduces an automated pipeline leveraging LLMs for mutant generation, demonstrating significant speed and quality improvements over baseline methods.

Findings

LLMs produce mutants up to 13x faster than manual methods.

LLMs generate fewer equivalent and duplicate mutants.

Few-shot prompting with low-to-medium temperature yields best results.

Abstract

Mutation analysis is a powerful technique for assessing test-suite adequacy, yet conventional approaches suffer from generating redundant, equivalent, or non-executable mutants. These challenges are particularly amplified in Simulink-Stateflow models due to the hierarchical structure these models have, which integrate continuous dynamics with discrete-event behaviors and are widely deployed in safety-critical Cyber-Physical Systems (CPSs). While prior work has explored machine learning and manually engineered mutation operators, these approaches remain constrained by limited training data and scalability issues. Motivated by recent advances in Large Language Models (LLMs), we investigate their potential to generate high-quality, domain-specific mutants for Simulink-Stateflow models. We develop an automated pipeline that converts Simulink-Stateflow models to structured JSON representations and systematically evaluates different mutation and prompting strategies across eight state-of-the-art LLMs. Through a comprehensive empirical study involving 38,400 LLM-generated mutants across four Simulink-Stateflow models, we demonstrate that LLMs generate mutants up to 13x faster than a manually engineered mutation-based baseline while producing significantly fewer equivalent and duplicate mutants and consistently achieving superior mutant quality. Moreover, our analysis reveals that few-shot prompting combined with low-to-medium temperature values yields optimal results. We provide an open-source prototype tool and release our complete dataset to facilitate reproducibility and advance future research in this domain.