Live Forensics for Distributed Storage Systems

TL;DR

Kaleidoscope is a live forensics system designed for large-scale distributed storage systems, enabling rapid root cause analysis of performance issues with minimal overhead.

Contribution

The paper introduces Kaleidoscope, a novel live forensics system that uses differential observability and stochastic modeling to identify root causes in distributed storage environments.

Findings

Pinpoints root causes of 95.8% of issues

Operates with 5-minute interval forensics

Imposes negligible monitoring overhead

Abstract

We present Kaleidoscope an innovative system that supports live forensics for application performance problems caused by either individual component failures or resource contention issues in large-scale distributed storage systems. The design of Kaleidoscope is driven by our study of I/O failures observed in a peta-scale storage system anonymized as PetaStore. Kaleidoscope is built on three key features: 1) using temporal and spatial differential observability for end-to-end performance monitoring of I/O requests, 2) modeling the health of storage components as a stochastic process using domain-guided functions that accounts for path redundancy and uncertainty in measurements, and, 3) observing differences in reliability and performance metrics between similar types of healthy and unhealthy components to attribute the most likely root causes. We deployed Kaleidoscope on PetaStore and our evaluation shows that Kaleidoscope can run live forensics at 5-minute intervals and pinpoint the root causes of 95.8% of real-world performance issues, with negligible monitoring overhead.