What It Would Take to Use Mutation Testing in Industry--A Study at Facebook
Moritz Beller, Chu-Pan Wong, Johannes Bader, Andrew Scott, Mateusz, Machalica, Satish Chandra, Erik Meijer

TL;DR
This study explores the feasibility of applying mutation testing at Facebook by learning error patterns and measuring test coverage, revealing that most mutants survive tests but some are useful for developers.
Contribution
It introduces a semi-automatic method to generate meaningful mutants based on real error patterns and operational anomalies, tailored for large-scale industrial systems.
Findings
Over 50% of generated mutants survive Facebook's test suite.
Developers find information from detected test holes interesting.
Nearly half of the developers would act on mutants by creating or updating tests.
Abstract
Traditionally, mutation testing generates an abundance of small deviations of a program, called mutants. At industrial systems the scale and size of Facebook's, doing this is infeasible. We should not create mutants that the test suite would likely fail on or that give no actionable signal to developers. To tackle this problem, in this paper, we semi-automatically learn error-inducing patterns from a corpus of common Java coding errors and from changes that caused operational anomalies at Facebook specifically. We combine the mutations with instrumentation that measures which tests exactly visited the mutated piece of code. Results on more than 15,000 generated mutants show that more than half of the generated mutants survive Facebook's rigorous test suite of unit, integration, and system tests. Moreover, in a case study with 26 developers, all but two found information of automatically…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
