Small is Beautiful: A Practical and Efficient Log Parsing Framework

TL;DR

EFParser is an unsupervised log parsing framework that enhances small LLMs' performance through architectural innovations, achieving state-of-the-art results efficiently.

Contribution

The paper introduces EFParser, a novel unsupervised log parser that improves small LLMs' effectiveness via a dual-cache system and validation module.

Findings

EFParser outperforms state-of-the-art baselines by 12.5% on average.

EFParser surpasses some large-scale model baselines with smaller models.

The framework maintains high efficiency despite additional validation steps.

Abstract

Log parsing is a fundamental step in log analysis, partitioning raw logs into constant templates and dynamic variables. While recent semantic-based parsers leveraging Large Language Models (LLMs) exhibit superior generalizability over traditional syntax-based methods, their effectiveness is heavily contingent on model scale. This dependency leads to significant performance collapse when employing smaller, more resource-efficient LLMs. Such degradation creates a major barrier to real-world adoption, where data privacy requirements and computational constraints necessitate the use of succinct models. To bridge this gap, we propose EFParser, an unsupervised LLM-based log parser designed to enhance the capabilities of smaller models through systematic architectural innovation. EFParser introduces a dual-cache system with an adaptive updating mechanism that distinguishes between novel patterns and variations of existing templates. This allows the parser to merge redundant templates and rectify prior errors, maintaining cache consistency. Furthermore, a dedicated correction module acts as a gatekeeper, validating and refining every LLM-generated template before caching to prevent error injection. Empirical evaluations on public large-scale datasets demonstrate that EFParser outperforms state-of-the-art baselines by an average of 12.5% across all metrics when running on smaller LLMs, even surpassing some baselines utilizing large-scale models. Despite its additional validation steps, EFParser maintains high computational efficiency, offering a robust and practical solution for real-world log analysis deployment.