CrossTrace: Efficient Cross-Thread and Cross-Service Span Correlation in Distributed Tracing for Microservices

TL;DR

CrossTrace introduces an efficient, zero-code distributed tracing method for microservices that accurately correlates spans across threads and services without source code modifications, using eBPF and a greedy inference algorithm.

Contribution

It proposes a novel zero-code tracing approach that leverages eBPF and a greedy algorithm for intra- and inter-service span correlation, reducing overhead and security risks.

Findings

Correlates thousands of spans within seconds

Achieves over 90% accuracy in span correlation

Supports large-scale microservice debugging

Abstract

Distributed tracing has become an essential technique for debugging and troubleshooting modern microservice-based applications, enabling software engineers to detect performance bottlenecks, identify failures, and gain insights into system behavior. However, implementing distributed tracing in large-scale applications remains challenging due to the need for extensive instrumentation. To reduce this burden, zero-code instrumentation solutions, such as those based on eBPF, have emerged, allowing span data to be collected without modifying application code. Despite this promise, span correlation, the process of establishing causal relationships between spans, remains a critical challenge in zero-code approaches. Existing solutions often rely on thread affinity, compromise system security by requiring the kernel integrity mode to be disabled, or incur significant computational overhead due to complex inference algorithms. This paper presents CrossTrace, a practical and efficient distributed tracing solution designed to support the debugging of microservice applications without requiring source code modifications. CrossTrace employs a greedy algorithm to infer intra-service span relationships from delay patterns, eliminating reliance on thread identifiers. For inter-service correlation, CrossTrace embeds span identifiers into TCP packet headers via eBPF, enabling secure and efficient correlation compromising system security policies. Evaluation results show that CrossTrace can correlate thousands of spans within seconds with over 90% accuracy, making it suitable for production deployment and valuable for microservice observability and diagnosis.