syslrn: Learning What to Monitor for Efficient Anomaly Detection

TL;DR

syslrn is a system that learns normal system behavior offline and then adapts online monitoring to improve anomaly detection efficiency with lower overhead.

Contribution

It introduces a novel approach to system monitoring by learning identifiers of normal behavior offline and customizing online monitoring, reducing overhead and improving detection.

Findings

Outperforms state-of-the-art log-analysis systems in a case study

Achieves detection with minimal overhead

Demonstrates effectiveness on OpenStack failure monitoring

Abstract

While monitoring system behavior to detect anomalies and failures is important, existing methods based on log-analysis can only be as good as the information contained in the logs, and other approaches that look at the OS-level software state introduce high overheads. We tackle the problem with syslrn, a system that first builds an understanding of a target system offline, and then tailors the online monitoring instrumentation based on the learned identifiers of normal behavior. While our syslrn prototype is still preliminary and lacks many features, we show in a case study for the monitoring of OpenStack failures that it can outperform state-of-the-art log-analysis systems with little overhead.