Simulink Mutation Testing using CodeBERT

TL;DR

This paper introduces BERTiMuS, a novel mutation testing approach for Simulink models using CodeBERT, which effectively generates mutants and outperforms existing tools in requirements-aware scenarios.

Contribution

BERTiMuS is the first method to leverage CodeBERT for Simulink mutation testing, demonstrating its effectiveness and complementarity to existing tools.

Findings

BERTiMuS can generate known Simulink mutation patterns.

It is complementary to FIM, a state-of-the-art tool.

Outperforms FIM in requirements-aware mutation testing.

Abstract

We present BERTiMuS, an approach that uses CodeBERT to generate mutants for Simulink models. BERTiMuS converts Simulink models into textual representations, masks tokens from the derived text, and uses CodeBERT to predict the masked tokens. Simulink mutants are obtained by replacing the masked tokens with predictions from CodeBERT. We evaluate BERTiMuS using Simulink models from an industrial benchmark, and compare it with FIM -- a state-of-the-art mutation tool for Simulink. We show that, relying exclusively on CodeBERT, BERTiMuS can generate the block-based Simulink mutation patterns documented in the literature. Further, our results indicate that: (a) BERTiMuS is complementary to FIM, and (b) when one considers a requirements-aware notion of mutation testing, BERTiMuS outperforms FIM.