Simulation-Driven Automated End-to-End Test and Oracle Inference

TL;DR

This paper presents ALPACAS, an automated tool that infers end-to-end integrity tests and their oracles at scale in industry, significantly improving test coverage and reliability for Meta's platforms.

Contribution

It introduces ALPACAS, the first scalable industrial system for automated inference of end-to-end tests and oracles based on production interventions.

Findings

ALPACAS increased test coverage compared to manual testing.

Automatically generated 39 production-ready tests from 3 million data points.

Test suite achieved an average pass rate of 99.84%, indicating low flakiness.

Abstract

This is the first work to report on inferential testing at scale in industry. Specifically, it reports the experience of automated testing of integrity systems at Meta. We built an internal tool called ALPACAS for automated inference of end-to-end integrity tests. Integrity tests are designed to keep users safe online by checking that interventions take place when harmful behaviour occurs on a platform. ALPACAS infers not only the test input, but also the oracle, by observing production interventions to prevent harmful behaviour. This approach allows Meta to automate the process of generating integrity tests for its platforms, such as Facebook and Instagram, which consist of hundreds of millions of lines of production code. We outline the design and deployment of ALPACAS, and report results for its coverage, number of tests produced at each stage of the test inference process, and their pass rates. Specifically, we demonstrate that using ALPACAS significantly improves coverage from a manual test design for the particular aspect of integrity end-to-end testing it was applied to. Further, from a pool of 3 million data points, ALPACAS automatically yields 39 production-ready end-to-end integrity tests. We also report that the ALPACAS-inferred test suite enjoys exceptionally low flakiness for end-to-end testing with its average in-production pass rate of 99.84%.