What It Would Take to Use Mutation Testing in Industry--A Study at Facebook

TL;DR

This study explores the feasibility of applying mutation testing at Facebook by learning error patterns and measuring test coverage, revealing that most mutants survive tests but some are useful for developers.

Contribution

It introduces a semi-automatic method to generate meaningful mutants based on real error patterns and operational anomalies, tailored for large-scale industrial systems.

Findings

Over 50% of generated mutants survive Facebook's test suite.

Developers find information from detected test holes interesting.

Nearly half of the developers would act on mutants by creating or updating tests.

Abstract

Traditionally, mutation testing generates an abundance of small deviations of a program, called mutants. At industrial systems the scale and size of Facebook's, doing this is infeasible. We should not create mutants that the test suite would likely fail on or that give no actionable signal to developers. To tackle this problem, in this paper, we semi-automatically learn error-inducing patterns from a corpus of common Java coding errors and from changes that caused operational anomalies at Facebook specifically. We combine the mutations with instrumentation that measures which tests exactly visited the mutated piece of code. Results on more than 15,000 generated mutants show that more than half of the generated mutants survive Facebook's rigorous test suite of unit, integration, and system tests. Moreover, in a case study with 26 developers, all but two found information of automatically detected test holes interesting in principle. As such, almost half of the 26 would actually act on the mutant presented to them by adapting an existing or creating a new test. The others did not for a variety of reasons often outside the scope of mutation testing. It remains a practical challenge how we can include such external information to increase the true actionability rate on mutants.