Transparently Capturing Request Execution Path for Anomaly Detection

TL;DR

This paper introduces REPTrace, a transparent method for capturing complete request execution paths in distributed systems, enabling effective anomaly detection with high accuracy and minimal overhead.

Contribution

It proposes a generic, transparent methodology for capturing end-to-end execution paths and applies it to anomaly detection in distributed platforms.

Findings

High recall (96%) in anomaly detection

Captures execution paths with reasonable latency

Negligible network overhead

Abstract

With the increasing scale and complexity of cloud systems and big data analytics platforms, it is becoming more and more challenging to understand and diagnose the processing of a service request in such distributed platforms. One way that helps to deal with this problem is to capture the complete end-to-end execution path of service requests among all involved components accurately. This paper presents REPTrace, a generic methodology for capturing such execution paths in a transparent fashion. We analyze a comprehensive list of execution scenarios, and propose principles and algorithms for generating the end-to-end request execution path for all the scenarios. Moreover, this paper presents an anomaly detection approach exploiting request execution paths to detect anomalies of the execution during request processing. The experiments on four popular distributed platforms with different workloads show that REPTrace can transparently capture the accurate request execution path with reasonable latency and negligible network overhead. Fault injection experiments show that execution anomalies are detected with high recall (96%).