Metric Criticality Identification for Cloud Microservices

TL;DR

KIMetrix is a data-driven system that automatically identifies minimal, critical metric subsets for effective monitoring of microservice architectures, reducing manual effort and improving reliability.

Contribution

We introduce KIMetrix, a novel information-theoretic approach that automatically selects essential metrics for microservice monitoring without requiring expert input or extensive logs.

Findings

Effectively reduces the number of metrics needed for monitoring.

Maintains comprehensive system coverage with fewer metrics.

Operates efficiently using lightweight metrics and traces.

Abstract

Modern cloud-native applications built on microservice architectures present unprecedented challenges for system monitoring and alerting. Site Reliability Engineers (SREs) face the daunting challenge of defining effective monitoring strategies across multitude of metrics to ensure system reliability, a task that traditionally requires extensive manual expertise. The distributed nature of microservices, characterized by stochastic execution patterns and intricate inter-service dependencies, renders the traditional manual approach of navigating the vast metrics landscape computationally and operationally prohibitive. To address this critical challenge, we propose KIMetrix, a data-driven system that automatically identifies minimal yet comprehensive metric subsets to aid SREs in monitoring microservice applications. KIMetrix leverages information-theoretic measures, specifically entropy and mutual information, to quantify metric criticality while considering the stochastic execution patterns inherent in microservice topologies. Our approach operates solely on lightweight metrics and traces, eliminating the need for expensive processing of unstructured logs, and requires no expert-defined training data. Experimental evaluation on state-of-the-art real-world microservice benchmark datasets demonstrates KIMetrix's effectiveness in identifying critical metric subsets that provide comprehensive system coverage while significantly reducing the burden on SREs. By automating the identification of essential metrics for alerting, KIMetrix enables more reliable system monitoring without overwhelming operators with false positives or missing critical system events.