Small-Amp: Test Amplification in a Dynamically Typed Language

TL;DR

This paper introduces a method for test amplification in dynamically typed languages by using profiling data to infer types and generate effective, understandable tests, achieving significant coverage improvements and human acceptance.

Contribution

It presents a novel approach to test amplification in dynamically typed languages using profiling to infer types, enabling effective test generation without explicit type info.

Findings

53% of test classes showed improved mutation coverage

72% of submitted pull requests were merged

Test amplification is feasible for dynamically typed languages

Abstract

Some test amplification tools extend a manually created test suite with additional test cases to increase the code coverage. The technique is effective, in the sense that it suggests strong and understandable test cases, generally adopted by software engineers. Unfortunately, the current state-of-the-art for test amplification heavily relies on program analysis techniques which benefit a lot from explicit type declarations present in statically typed languages. In dynamically typed languages, such type declarations are not available and as a consequence test amplification has yet to find its way to programming languages like Smalltalk, Python, Ruby and Javascript. We propose to exploit profiling information --readily obtainable by executing the associated test suite-- to infer the necessary type information creating special test inputs with corresponding assertions. We evaluated this approach on 52 selected test classes from 13 mature projects in the Pharo ecosystem containing approximately 400 test methods. We show the improvement in killing new mutants and mutation coverage at least in 28 out of 52 test classes (53%). Moreover, these generated tests are understandable by humans: 8 out of 11 pull-requests submitted were merged into the main code base (72%). These results are comparable to the state-of-the-art, hence we conclude that test amplification is feasible for dynamically typed languages.