Anomaly Detection for Incident Response at Scale

TL;DR

This paper introduces AIDR, a machine learning-based anomaly detection system deployed at Walmart that improves incident detection speed and accuracy by combining statistical, ML, and rule-based methods, with scalable, real-time monitoring.

Contribution

The paper presents a scalable, real-time anomaly detection system integrating multiple ML models and rule-based thresholds, with feedback mechanisms for continuous improvement, tailored for large-scale enterprise use.

Findings

Detected 63% of major incidents

Reduced mean-time-to-detect by over 7 minutes

Lowered false positive rate compared to previous methods

Abstract

We present a machine learning-based anomaly detection product, AI Detect and Respond (AIDR), that monitors Walmart's business and system health in real-time. During the validation over 3 months, the product served predictions from over 3000 models to more than 25 application, platform, and operation teams, covering 63\% of major incidents and reducing the mean-time-to-detect (MTTD) by more than 7 minutes. Unlike previous anomaly detection methods, our solution leverages statistical, ML and deep learning models while continuing to incorporate rule-based static thresholds to incorporate domain-specific knowledge. Both univariate and multivariate ML models are deployed and maintained through distributed services for scalability and high availability. AIDR has a feedback loop that assesses model quality with a combination of drift detection algorithms and customer feedback. It also offers self-onboarding capabilities and customizability. AIDR has achieved success with various internal teams with lower time to detection and fewer false positives than previous methods. As we move forward, we aim to expand incident coverage and prevention, reduce noise, and integrate further with root cause recommendation (RCR) to enable an end-to-end AIDR experience.