On Sequence-to-Sequence Models for Automated Log Parsing

TL;DR

This paper systematically evaluates sequence-to-sequence models for automated log parsing, comparing architectures like Transformer, Mamba, and LSTMs to determine their accuracy, efficiency, and practical implications.

Contribution

It provides a comprehensive empirical comparison of different sequence modelling architectures for log parsing, highlighting the effectiveness of Transformers and Mamba in accuracy and efficiency.

Findings

Transformers achieve the lowest parsing error.

Mamba offers competitive accuracy with lower computational cost.

Character-level tokenization improves performance.

Abstract

Log parsing is a critical standard operating procedure in software systems, enabling monitoring, anomaly detection, and failure diagnosis. However, automated log parsing remains challenging due to heterogeneous log formats, distribution shifts between training and deployment data, and the brittleness of rule-based approaches. This study aims to systematically evaluate how sequence modelling architecture, representation choice, sequence length, and training data availability influence automated log parsing performance and computational cost. We conduct a controlled empirical study comparing four sequence modelling architectures: Transformer, Mamba state-space, monodirectional LSTM, and bidirectional LSTM models. In total, 396 models are trained across multiple dataset configurations and evaluated using relative Levenshtein edit distance with statistical significance testing. Transformer achieves the lowest mean relative edit distance (0.111), followed by Mamba (0.145), mono-LSTM (0.186), and bi-LSTM (0.265), where lower values are better. Mamba provides competitive accuracy with substantially lower computational cost. Character-level tokenization generally improves performance, sequence length has negligible practical impact on Transformer accuracy, and both Mamba and Transformer demonstrate stronger sample efficiency than recurrent models. Overall, Transformers reduce parsing error by 23.4%, while Mamba is a strong alternative under data or compute constraints. These results also clarify the roles of representation choice, sequence length, and sample efficiency, providing practical guidance for researchers and practitioners.