Scalable Statistical Root Cause Analysis on App Telemetry

TL;DR

Minesweeper is a scalable, automated root cause analysis technique that efficiently identifies bug causes from symptoms in Facebook's app telemetry data, significantly reducing manual effort and improving accuracy.

Contribution

The paper introduces Minesweeper, a novel scalable algorithm that automatically mines patterns from telemetry data to accurately identify root causes of bugs at Facebook scale.

Findings

Performs RCA on tens of thousands of reports in under 3 minutes.

Achieves over 85% accuracy in root cause identification.

Effectively handles diverse symptoms across billions of user environments.

Abstract

Despite engineering workflows that aim to prevent buggy code from being deployed, bugs still make their way into the Facebook app. When symptoms of these bugs, such as user submitted reports and automatically captured crashes, are reported, finding their root causes is an important step in resolving them. However, at Facebook's scale of billions of users, a single bug can manifest as several different symptoms according to the various user and execution environments in which the software is deployed. Root cause analysis (RCA) therefore requires tedious manual investigation and domain expertise to extract out common patterns that are observed in groups of reports and use them for debugging. We propose Minesweeper, a technique for RCA that moves towards automatically identifying the root cause of bugs from their symptoms. The method is based on two key aspects: (i) a scalable algorithm to efficiently mine patterns from telemetric information that is collected along with the reports, and (ii) statistical notions of precision and recall of patterns that help point towards root causes. We evaluate Minesweeper's scalability and effectiveness in finding root causes from symptoms on real world bug and crash reports from Facebook's apps. Our evaluation demonstrates that Minesweeper can perform RCA for tens of thousands of reports in less than 3 minutes, and is more than 85% accurate in identifying the root cause of regressions.