A Survey on Automated Log Analysis for Reliability Engineering

TL;DR

This survey comprehensively reviews automated log analysis techniques used in reliability engineering, covering methods for log parsing, anomaly detection, failure prediction, and the use of open-source tools and datasets, highlighting future research directions.

Contribution

It provides a detailed overview of recent advances in automated log analysis, including techniques, tools, datasets, and future research directions in reliability engineering.

Findings

Automated log analysis enhances reliability engineering processes.

Open-source tools and datasets facilitate research and application.

Future directions include real-world deployment and next-generation techniques.

Abstract

Logs are semi-structured text generated by logging statements in software source code. In recent decades, software logs have become imperative in the reliability assurance mechanism of many software systems because they are often the only data available that record software runtime information. As modern software is evolving into a large scale, the volume of logs has increased rapidly. To enable effective and efficient usage of modern software logs in reliability engineering, a number of studies have been conducted on automated log analysis. This survey presents a detailed overview of automated log analysis research, including how to automate and assist the writing of logging statements, how to compress logs, how to parse logs into structured event templates, and how to employ logs to detect anomalies, predict failures, and facilitate diagnosis. Additionally, we survey work that releases open-source toolkits and datasets. Based on the discussion of the recent advances, we present several promising future directions toward real-world and next-generation automated log analysis.