What Distributed Systems Say: A Study of Seven Spark Application Logs

TL;DR

This study analyzes how different logging verbosity levels affect execution time, storage, and diagnostic effectiveness in Spark applications, providing insights for optimizing logging practices in distributed systems.

Contribution

It presents an experimental evaluation of logging impacts on performance and log usefulness across multiple Spark benchmarks and failure scenarios.

Findings

Higher verbosity increases storage and overhead.

Optimal verbosity balances detail and performance.

Logs provide valuable insights for failure diagnosis.

Abstract

Execution logs are a crucial medium as they record runtime information of software systems. Although extensive logs are helpful to provide valuable details to identify the root cause in postmortem analysis in case of a failure, this may also incur performance overhead and storage cost. Therefore, in this research, we present the result of our experimental study on seven Spark benchmarks to illustrate the impact of different logging verbosity levels on the execution time and storage cost of distributed software systems. We also evaluate the log effectiveness and the information gain values, and study the changes in performance and the generated logs for each benchmark with various types of distributed system failures. Our research draws insightful findings for developers and practitioners on how to set up and utilize their distributed systems to benefit from the execution logs.