LogPrism: Unifying Structure and Variable Encoding for Effective Log Compression

TL;DR

LogPrism introduces a unified framework that combines structural and variable encoding for log compression, significantly improving compression ratios and processing speed by capturing deep redundancies without relying on rigid pre-parsing.

Contribution

It proposes a novel hierarchical approach using a Unified Redundancy Tree to dynamically encode structure and variables jointly, surpassing existing methods in efficiency and effectiveness.

Findings

Achieves highest compression ratio on 14 out of 16 datasets.

Surpasses baselines by 6.12% to 83.34% in compression.

Delivers throughput of 29.87 MB/s, 1.68× to 43.04× faster than competitors.

Abstract

In the field of log compression, the prevailing "parse-then-compress" paradigm fundamentally limits effectiveness by treating log parsing and compression as isolated objectives. While parsers prioritize semantic accuracy (i.e., event identification), they often obscure deep correlations between static templates and dynamic variables that are critical for storage efficiency. In this paper, we investigate this misalignment through a comprehensive empirical study and propose LogPrism, a framework that bridges the gap via unified redundancy encoding. Rather than relying on a rigid pre-parsing step, LogPrism dynamically integrates structural extraction with variable encoding by constructing a Unified Redundancy Tree (URT). This hierarchical approach effectively mines "structure+variable" co-occurrence patterns, capturing deep contextual redundancies while accelerating processing through pre-emptive pattern encoding. Extensive experiments on 16 benchmark datasets confirm that LogPrism establishes a new state-of-the-art. It achieves the highest compression ratio on 14 datasets, surpassing existing baselines by margins of 6.12% to 83.34%, while delivering superior throughput at 29.87 MB/s (1.68$\times$~43.04$\times$ faster than competitors). Moreover, when configured in single-archive mode to maximize global pattern discovery, LogPrism boosts its compression ratio by 273.27%, outperforming the best baseline by 19.39% with a 2.62$\times$ speed advantage.