Impact of Log Parsing on Deep Learning-Based Anomaly Detection

TL;DR

This study empirically examines how different log parsing techniques influence deep learning-based anomaly detection accuracy, revealing that distinguishability rather than parsing accuracy itself is crucial for effective detection.

Contribution

It provides a comprehensive empirical analysis of 13 log parsing methods across multiple anomaly detection techniques and datasets, highlighting the importance of distinguishability over parsing accuracy.

Findings

No strong correlation between log parsing accuracy and anomaly detection accuracy.

Distinguishability of log parsing results is key to effective anomaly detection.

Empirical validation of theoretical results on the role of distinguishability.

Abstract

Software systems log massive amounts of data, recording important runtime information. Such logs are used, for example, for log-based anomaly detection, which aims to automatically detect abnormal behaviors of the system under analysis by processing the information recorded in its logs. Many log-based anomaly detection techniques based on deep learning models include a pre-processing step called log parsing. However, understanding the impact of log parsing on the accuracy of anomaly detection techniques has received surprisingly little attention so far. Investigating what are the key properties log parsing techniques should ideally have to help anomaly detection is therefore warranted. In this paper, we report on a comprehensive empirical study on the impact of log parsing on anomaly detection accuracy, using 13 log parsing techniques, seven anomaly detection techniques (five based on deep learning and two based on traditional machine learning) on three publicly available log datasets. Our empirical results show that, despite what is widely assumed, there is no strong correlation between log parsing accuracy and anomaly detection accuracy, regardless of the metric used for measuring log parsing accuracy. Moreover, we experimentally confirm existing theoretical results showing that it is a property that we refer to as distinguishability in log parsing results as opposed to their accuracy that plays an essential role in achieving accurate anomaly detection.