LLMParser: An Exploratory Study on Using Large Language Models for Log Parsing

TL;DR

This paper investigates the use of large language models for log parsing, demonstrating that LLMs can outperform traditional parsers in accuracy, with smaller models sometimes being more effective and pre-training effects being inconsistent.

Contribution

The study introduces LLMParser, an LLM-based log parser, and provides empirical analysis on model size, training data, and pre-training effects, highlighting limitations and future directions.

Findings

LLMParser achieves 96% parsing accuracy, outperforming state-of-the-art parsers.

Smaller LLMs like Flan-T5-base can match larger models with faster inference.

Pre-training on logs does not always improve parsing accuracy, sometimes decreasing it.

Abstract

Logs are important in modern software development with runtime information. Log parsing is the first step in many log-based analyses, that involve extracting structured information from unstructured log data. Traditional log parsers face challenges in accurately parsing logs due to the diversity of log formats, which directly impacts the performance of downstream log-analysis tasks. In this paper, we explore the potential of using Large Language Models (LLMs) for log parsing and propose LLMParser, an LLM-based log parser based on generative LLMs and few-shot tuning. We leverage four LLMs, Flan-T5-small, Flan-T5-base, LLaMA-7B, and ChatGLM-6B in LLMParsers. Our evaluation of 16 open-source systems shows that LLMParser achieves statistically significantly higher parsing accuracy than state-of-the-art parsers (a 96% average parsing accuracy). We further conduct a comprehensive empirical analysis on the effect of training size, model size, and pre-training LLM on log parsing accuracy. We find that smaller LLMs may be more effective than more complex LLMs; for instance where Flan-T5-base achieves comparable results as LLaMA-7B with a shorter inference time. We also find that using LLMs pre-trained using logs from other systems does not always improve parsing accuracy. While using pre-trained Flan-T5-base shows an improvement in accuracy, pre-trained LLaMA results in a decrease (decrease by almost 55% in group accuracy). In short, our study provides empirical evidence for using LLMs for log parsing and highlights the limitations and future research direction of LLM-based log parsers.