Causal AI-based Root Cause Identification: Research to Practice at Scale

TL;DR

This paper introduces a causality-based Root Cause Identification algorithm integrated into IBM Instana, enabling real-time failure diagnosis in large distributed systems to improve reliability and performance.

Contribution

The paper presents a novel causality-focused RCI algorithm and demonstrates its successful deployment in IBM Instana for enterprise-scale, real-time system failure diagnosis.

Findings

Enhanced failure diagnosis accuracy in large systems

Real-time root cause identification in production environments

Improved system reliability and performance metrics

Abstract

Modern applications are built as large, distributed systems spanning numerous modules, teams, and data centers. Despite robust engineering and recovery strategies, failures and performance issues remain inevitable, risking significant disruptions and affecting end users. Rapid and accurate root cause identification is therefore vital to ensure system reliability and maintain key service metrics. We have developed a novel causality-based Root Cause Identification (RCI) algorithm that emphasizes causation over correlation. This algorithm has been integrated into IBM Instana-bridging research to practice at scale-and is now in production use by enterprise customers. By leveraging "causal AI," Instana stands apart from typical Application Performance Management (APM) tools, pinpointing issues in near real-time. This paper highlights Instana's advanced failure diagnosis capabilities, discussing both the theoretical underpinnings and practical implementations of the RCI algorithm. Real-world examples illustrate how our causality-based approach enhances reliability and performance in today's complex system landscapes.