LogFold: Compressing Logs with Structured Tokens and Hybrid Encoding

TL;DR

LogFold introduces a novel log compression method that leverages structured token analysis and hybrid encoding to significantly outperform existing algorithms in compression ratio and speed.

Contribution

This paper presents LogFold, a new log compression approach that exploits redundancies in structured tokens and employs a fine-grained, type-aware encoding strategy.

Findings

Achieves 11.11% better compression ratios than state-of-the-art methods.

Operates at a speed of 9.842 MB/s on average.

Component ablation confirms the effectiveness of each part.

Abstract

Logs are essential for diagnosing failures and conducting retrospective studies, leading many software organizations to retain log messages for a long time. Nevertheless, the volume of generated log data grows rapidly as software systems grow, necessitating an effective compression method. Apart from general-purpose compressors (e.g., Gzip, Bzip2), many recent studies developed log-specific compression algorithms, but they offer suboptimal performance because of (1) overlooking redundancies within certain complex tokens, and (2) lacking a fine-grained encoding strategy for diverse token types. This work uncovers a new redundancy pattern in structured tokens and proposes a new type-aware encoding strategy to improve log compression. Building on this insight, we introduce LogFold, a novel log compression method consisting of four components: a token analyzer to classifies tokens as structured, unstructured, or static types; a processor that mines recurring patterns within structured tokens based on their delimiter skeletons; a hybrid encoder that tailors data representation according to token types; and a packer that compresses the output into an archive file. Extensive experiments on 16 public log datasets demonstrate that LogFold surpasses state-of-the-art baselines, achieving average compression ratio improvements by 11.11%, with a compression speed of 9.842 MB/s. Ablation studies further indicate the importance of each component. We also conduct sensitivity analyses to verify LogFold's robustness and stability across various internal settings.