Lost in Translation? Converting RegExes for Log Parsing into Dynatrace Pattern Language

TL;DR

This paper introduces Reptile, a tool that automates converting regular expressions into Dynatrace Pattern Language for log parsing, using rule-based methods and GPT-4 optimization, to facilitate migration and improve accuracy.

Contribution

Reptile combines rule-based conversion with GPT-4 optimization to automate and enhance the translation of RegExes into DPL, addressing manual conversion challenges.

Findings

Reptile safely converted 73.7% of 946 RegExes.

Pattern optimization achieved F1-score and MCC above 0.91.

Reptile offers practical benefits for migrating to Dynatrace.

Abstract

Log files provide valuable information for detecting and diagnosing problems in enterprise software applications and data centers. Several log analytics tools and platforms were developed to help filter and extract information from logs, typically using regular expressions (RegExes). Recent commercial log analytics platforms provide domain-specific languages specifically designed for log parsing, such as Grok or the Dynatrace Pattern Language (DPL). However, users who want to migrate to these platforms must manually convert their RegExes into the new pattern language, which is costly and error-prone. In this work, we present Reptile, which combines a rule-based approach for converting RegExes into DPL patterns with a best-effort approach for cases where a full conversion is impossible. Furthermore, it integrates GPT-4 to optimize the obtained DPL patterns. The evaluation with 946 RegExes collected from a large company shows that Reptile safely converted 73.7% of them. The evaluation of Reptile's pattern optimization with 23 real-world RegExes showed an F1-score and MCC above 0.91. These results are promising and have ample practical implications for companies that migrate to a modern log analytics platform, such as Dynatrace.