KELP: Robust Online Log Parsing Through Evolutionary Grouping Trees

TL;DR

KELP is a novel online log parser that uses evolutionary grouping trees to adaptively and accurately parse logs in dynamic production environments, outperforming static template-based methods.

Contribution

We introduce KELP, a high-throughput online log parser with a new evolutionary grouping tree data structure that adapts to schema drifts in real-time, and a benchmark reflecting production complexity.

Findings

KELP maintains high accuracy on complex, realistic datasets.

Traditional heuristic parsers fail under production-like log variability.

KELP achieves robust, real-time parsing without sacrificing throughput.

Abstract

Real-time log analysis is the cornerstone of observability for modern infrastructure. However, existing online parsers are architecturally unsuited for the dynamism of production environments. Built on fundamentally static template models, they are dangerously brittle: minor schema drifts silently break parsing pipelines, leading to lost alerts and operational toil. We propose \textbf{KELP} (\textbf{K}elp \textbf{E}volutionary \textbf{L}og \textbf{P}arser), a high-throughput parser built on a novel data structure: the Evolutionary Grouping Tree. Unlike heuristic approaches that rely on fixed rules, KELP treats template discovery as a continuous online clustering process. As logs arrive, the tree structure evolves, nodes split, merge, and re-evaluate roots based on changing frequency distributions. Validating this adaptability requires a dataset that models realistic production complexity, yet we identify that standard benchmarks rely on static, regex-based ground truths that fail to reflect this. To enable rigorous evaluation, we introduce a new benchmark designed to reflect the structural ambiguity of modern production systems. Our evaluation demonstrates that KELP maintains high accuracy on this rigorous dataset where traditional heuristic methods fail, without compromising throughput. Our code and dataset can be found at codeberg.org/stonebucklabs/kelp