Experience Report: Deep Learning-based System Log Analysis for Anomaly Detection

TL;DR

This paper provides a comprehensive review and evaluation of deep learning-based log anomaly detection methods, comparing five neural network models across large datasets to guide future research and industrial use.

Contribution

It offers the first rigorous comparison of neural network-based log anomaly detectors, highlighting their characteristics and performance on large-scale datasets.

Findings

Deep learning methods outperform traditional approaches in anomaly detection.

Supervised models show higher accuracy than unsupervised ones.

Evaluation on large datasets reveals strengths and limitations of each method.

Abstract

Logs have been an imperative resource to ensure the reliability and continuity of many software systems, especially large-scale distributed systems. They faithfully record runtime information to facilitate system troubleshooting and behavior understanding. Due to the large scale and complexity of modern software systems, the volume of logs has reached an unprecedented level. Consequently, for log-based anomaly detection, conventional manual inspection methods or even traditional machine learning-based methods become impractical, which serve as a catalyst for the rapid development of deep learning-based solutions. However, there is currently a lack of rigorous comparison among the representative log-based anomaly detectors that resort to neural networks. Moreover, the re-implementation process demands non-trivial efforts, and bias can be easily introduced. To better understand the characteristics of different anomaly detectors, in this paper, we provide a comprehensive review and evaluation of five popular neural networks used by six state-of-the-art methods. Particularly, four of the selected methods are unsupervised, and the remaining two are supervised. These methods are evaluated with two publicly available log datasets, which contain nearly 16 million log messages and 0.4 million anomaly instances in total. We believe our work can serve as a basis in this field and contribute to future academic research and industrial applications.