Hybrid Fault-Driven Mutation Testing for Python

TL;DR

This paper introduces PyTation, a mutation testing tool for Python that uses novel fault-inspired operators and a hybrid analysis approach to generate diverse mutants, improving fault detection over existing tools.

Contribution

The paper presents seven new mutation operators tailored for Python, a hybrid static-dynamic analysis method, and a tool that enhances mutation testing effectiveness for dynamically typed languages.

Findings

PyTation generates mutants that complement existing tools.

It uncovers faults not detected by general-purpose mutation operators.

PyTation produces a high proportion of unique mutants with low cross-kill rate.

Abstract

Mutation testing is an effective technique for assessing the effectiveness of test suites by systematically injecting artificial faults into programs. However, existing mutation testing techniques fall short in capturing many types of common faults in dynamically typed languages like Python. In this paper, we introduce a novel set of seven mutation operators that are inspired by prevalent anti-patterns in Python programs, designed to complement the existing general-purpose operators and broaden the spectrum of simulated faults. We propose a mutation testing technique that utilizes a hybrid of static and dynamic analyses to mutate Python programs based on these operators while minimizing equivalent mutants. We implement our approach in a tool called PyTation and evaluate it on 13 open-source Python applications. Our results show that PyTation generates mutants that complement those from general-purpose tools, exhibiting distinct behaviour under test execution and uncovering inadequacies in high-coverage test suites. We further demonstrate that PyTation produces a high proportion of unique mutants, a low cross-kill rate, and a low test overlap ratio relative to baseline tools, highlighting its novel fault model. PyTation also incurs few equivalent mutants, aided by dynamic analysis heuristics.