MicLog: Towards Accurate and Efficient LLM-based Log Parsing via Progressive Meta In-Context Learning

TL;DR

MicLog introduces a progressive meta in-context learning framework that significantly improves the accuracy and efficiency of LLM-based log parsing by combining meta-learning with dynamic example selection and caching strategies.

Contribution

This paper presents the first ProgMeta-ICL framework for log parsing, enhancing LLM capabilities and reducing parsing time through novel meta-learning and caching techniques.

Findings

Achieves 10.3% higher accuracy than state-of-the-art

Reduces parsing time by 42.4%

Effectively handles semantic variations in logs

Abstract

Log parsing converts semi-structured logs into structured templates, forming a critical foundation for downstream analysis. Traditional syntax and semantic-based parsers often struggle with semantic variations in evolving logs and data scarcity stemming from their limited domain coverage. Recent large language model (LLM)-based parsers leverage in-context learning (ICL) to extract semantics from examples, demonstrating superior accuracy. However, LLM-based parsers face two main challenges: 1) underutilization of ICL capabilities, particularly in dynamic example selection and cross-domain generalization, leading to inconsistent performance; 2) time-consuming and costly LLM querying. To address these challenges, we present MicLog, the first progressive meta in-context learning (ProgMeta-ICL) log parsing framework that combines meta-learning with ICL on small open-source LLMs (i.e., Qwen-2.5-3B). Specifically, MicLog: i) enhances LLMs' ICL capability through a zero-shot to k-shot ProgMeta-ICL paradigm, employing weighted DBSCAN candidate sampling and enhanced BM25 demonstration selection; ii) accelerates parsing via a multi-level pre-query cache that dynamically matches and refines recently parsed templates. Evaluated on Loghub-2.0, MicLog achieves 10.3% higher parsing accuracy than the state-of-the-art parser while reducing parsing time by 42.4%.