Stronger, Cheaper and Demonstration-Free Log Parsing with LLMs
Yi Xiao, Van-Hoang Le, Hongyu Zhang

TL;DR
This paper introduces LogBatcher, a cost-effective, demonstration-free log parser using LLMs that partitions logs, matches templates via caching, and batches logs for improved efficiency across diverse datasets.
Contribution
LogBatcher is a novel LLM-based log parser that eliminates the need for training data and demonstrations, reducing overhead through clustering and caching techniques.
Findings
Effective across 16 public datasets
Reduces LLM invocation overhead
Achieves accurate log parsing results
Abstract
Log parsing, the process of converting raw log messages into structured formats, is an important initial step for automated analysis of logs of large-scale software systems. Traditional log parsers often rely on heuristics or handcrafted features, which may not generalize well across diverse log sources or require extensive model tuning. Recently, some log parsers have utilized powerful generative capabilities of large language models (LLMs). However, they heavily rely on demonstration examples, resulting in substantial overhead in LLM invocations. To address these issues, we propose LogBatcher, a cost-effective LLM-based log parser that requires no training process or labeled data. To leverage latent characteristics of log data and reduce the overhead, we divide logs into several partitions through clustering. Then we perform a cache matching process to match logs with previously…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsNatural Language Processing Techniques · Speech Recognition and Synthesis · Network Packet Processing and Optimization
