A Comparative Study on Large Language Models for Log Parsing

TL;DR

This study evaluates the effectiveness of various large language models, including free and proprietary ones, in automating log parsing tasks across multiple open-source projects, revealing that smaller free models can perform competitively.

Contribution

It provides a comparative analysis of six recent LLMs for log parsing, highlighting the potential of free-to-use models in this domain.

Findings

Free models like CodeLlama outperform GPT-3.5 in log template extraction.

Free-to-use models are comparable to paid models in log parsing accuracy.

Smaller, code-specialized models show significant promise for log parsing tasks.

Abstract

Background: Log messages provide valuable information about the status of software systems. This information is provided in an unstructured fashion and automated approaches are applied to extract relevant parameters. To ease this process, log parsing can be applied, which transforms log messages into structured log templates. Recent advances in language models have led to several studies that apply ChatGPT to the task of log parsing with promising results. However, the performance of other state-of-the-art large language models (LLMs) on the log parsing task remains unclear. Aims: In this study, we investigate the current capability of state-of-the-art LLMs to perform log parsing. Method: We select six recent LLMs, including both paid proprietary (GPT-3.5, Claude 2.1) and four free-to-use open models, and compare their performance on system logs obtained from a selection of mature open-source projects. We design two different prompting approaches and apply the LLMs on 1, 354 log templates across 16 different projects. We evaluate their effectiveness, in the number of correctly identified templates, and the syntactic similarity between the generated templates and the ground truth. Results: We found that free-to-use models are able to compete with paid models, with CodeLlama extracting 10% more log templates correctly than GPT-3.5. Moreover, we provide qualitative insights into the usability of language models (e.g., how easy it is to use their responses). Conclusions: Our results reveal that some of the smaller, free-to-use LLMs can considerably assist log parsing compared to their paid proprietary competitors, especially code-specialized models.