LLMLog: Advanced Log Template Generation via LLM-driven Multi-Round Annotation

TL;DR

LLMLog is a multi-round annotation framework that leverages adaptive in-context learning and similarity metrics to improve log template generation accuracy using large language models.

Contribution

It introduces a novel multi-round annotation method with adaptive context selection and a similarity metric to enhance LLM-based log template generation.

Findings

Outperforms state-of-the-art methods on sixteen datasets.

Effective in handling complex and ambiguous log content.

Improves template accuracy through adaptive context and selection strategies.

Abstract

Modern computing systems, such as HDFS and Spark, produce vast quantities of logs that developers use for tasks like anomaly detection and error analysis. To simplify log analysis, template generation methods have been proposed to standardize log formats, transforming unstructured data into structured templates. Existing heuristic-based methods and neural network-based methods suffer from low accuracy problems due to the reliance on handcrafted heuristics or specific log patterns in training sets. Recently, large language models (LLMs) have shown great potential in log template generation. However, they often struggle with ambiguous, complex, or highly specific log content, which can lead to errors in generating accurate templates. To address these challenges, we propose LLMLog, a multi-round annotation framework with adaptive in-context learning. We first propose an edit-distance-based similarity metric to evaluate log similarity. Then, we introduce a method to select the most informative $k$ unlabeled logs for annotation by considering both the representativeness of the logs and the confidence of LLM predictions. Additionally, we design an adaptive context selection strategy that adaptively selects labeled logs to ensure comprehensive keyword coverage for unlabeled logs. These labeled logs serve as the context for LLMs to better understand the unlabeled logs, thereby enhancing the accuracy of template generation. Extensive experiments on sixteen datasets demonstrate that LLMLog outperforms the state-of-the-art approaches.