MR-Adopt: Automatic Deduction of Input Transformation Function for Metamorphic Testing

TL;DR

MR-Adopt automatically deduces input transformations from hard-coded test cases using LLMs, enabling reuse of metamorphic relations to improve test coverage and mutation scores.

Contribution

It introduces an LLM-based approach to automatically generate input transformations from encoded MRs, enhancing test case reusability and effectiveness.

Findings

MR-Adopt applies to 72% of encoded MRs, outperforming vanilla GPT-3.5 by 33.33%.

Test cases with MR-Adopt increase line coverage by 10.62%.

Mutation score improves by 18.91% with MR-Adopt transformations.

Abstract

While a recent study reveals that many developer-written test cases can encode a reusable Metamorphic Relation (MR), over 70% of them directly hard-code the source input and follow-up input in the encoded relation. Such encoded MRs, which do not contain an explicit input transformation to transform the source inputs to corresponding follow-up inputs, cannot be reused with new source inputs to enhance test adequacy. In this paper, we propose MR-Adopt (Automatic Deduction Of inPut Transformation) to automatically deduce the input transformation from the hard-coded source and follow-up inputs, aiming to enable the encoded MRs to be reused with new source inputs. With typically only one pair of source and follow-up inputs available in an MR-encoded test case as the example, we leveraged LLMs to understand the intention of the test case and generate additional examples of source-followup input pairs. This helps to guide the generation of input transformations generalizable to multiple source inputs. Besides, to mitigate the issue that LLMs generate erroneous code, we refine LLM-generated transformations by removing MR- irrelevant code elements with data-flow analysis. Finally, we assess candidate transformations based on encoded output relations and select the best transformation as the result. Evaluation results show that MR-Adopt can generate input transformations applicable to all experimental source inputs for 72.00% of encoded MRs, which is 33.33% more than using vanilla GPT-3.5. By incorporating MR- Adopt-generated input transformations, encoded MR-based test cases can effectively enhance the test adequacy, increasing the line coverage and mutation score by 10.62% and 18.91%, respectively.