# Delog: A Privacy Preserving Log Filtering Framework for Online Compute Platforms

## TL;DR

Delog is a privacy-preserving log filtering framework for online platforms that uses distributed parsing with Locality Sensitive Hashing to improve error detection and log relevance, demonstrating scalability and superior performance.

## Contribution

The paper introduces Delog, a novel privacy-preserving log filtering framework with a scalable distributed parsing algorithm leveraging LSH, outperforming existing methods.

## Key findings

- Outperforms state-of-the-art log parsing methods.
- Scales effectively to large datasets with millions of log lines.
- Maintains privacy of user logs while enabling useful log analysis.

## Abstract

In many software applications, logs serve as the only interface between the application and the developer. However, navigating through the logs of long-running applications is often challenging. Logs from previously successful application runs can be leveraged to automatically identify errors and provide users with only the logs that are relevant to the debugging process. We describe a privacy preserving framework which can be employed by Platform as a Service (PaaS) providers to utilize the user logs generated on the platform while protecting the potentially sensitive logged data. Further, in order to accurately and scalably parse log lines, we present a distributed log parsing algorithm which leverages Locality Sensitive Hashing (LSH). We outperform the state-of-the-art on multiple datasets. We further demonstrate the scalability of Delog on publicly available Thunderbird log dataset with close to 27,000 unique patterns and 211 million lines.