Run-time Failure Detection via Non-intrusive Event Analysis in a Large-Scale Cloud Computing Platform

TL;DR

This paper presents a non-intrusive run-time failure detection method for large-scale cloud systems that uses event analysis to identify silent failures more effectively than existing logging approaches.

Contribution

It introduces a lightweight, non-intrusive event tracing approach that builds fault-free monitoring rules without system modifications, improving failure detection in cloud platforms.

Findings

Detects failures with an F1 score of 0.85, outperforming existing mechanisms.

Reduces failure detection time to approximately 114 seconds.

Achieves higher accuracy (0.77) compared to traditional logging (0.50).

Abstract

Cloud computing systems fail in complex and unforeseen ways due to unexpected combinations of events and interactions among hardware and software components. These failures are especially problematic when they are silent, i.e., not accompanied by any explicit failure notification, hindering the timely detection and recovery. In this work, we propose an approach to run-time failure detection tailored for monitoring multi-tenant and concurrent cloud computing systems. The approach uses a non-intrusive form of event tracing, without manual changes to the system's internals to propagate session identifiers (IDs), and builds a set of lightweight monitoring rules from fault-free executions. We evaluated the effectiveness of the approach in detecting failures in the context of the OpenStack cloud computing platform, a complex and "off-the-shelf" distributed system, by executing a campaign of fault injection experiments in a multi-tenant scenario. Our experiments show that the approach detects the failure with an F1 score (0.85) and accuracy (0.77) higher than the ones provided by the OpenStack failure logging mechanisms (0.53 and 0.50) and two non--session-aware run-time verification approaches (both lower than 0.15). Moreover, the approach significantly decreases the average time to detect failures at run-time (~114 seconds) compared to the OpenStack logging mechanisms.