Evaluating Design Conformance Through Trace Comparison

TL;DR

This paper presents a method to quantitatively evaluate how closely distributed system implementations adhere to their original design models by comparing runtime traces to design traces using conformance checking.

Contribution

It introduces a conformance checking approach for distributed systems that quantifies adherence to design models using industry-standard OpenTelemetry traces.

Findings

Conformance percentage effectively measures design adherence.

Method applicable to a wide range of distributed systems.

Tracks divergence over time to maintain system coherence.

Abstract

The design of a system and its implementation are two tasks often carried out by different individuals on a development team, and can occur weeks or months apart. This creates a potential for divergence between real behavior and the designed model that an implementation is intended to match. Particularly as time passes and individuals who were present for the original conception of the design leave, a system can lose coherence and drift from intended design principles. Even with a robust system design, more is needed to ensure that the key implementation details match the design and that adherence to a particular strategy is not lost over time. This paper proposes an approach to address that concern for distributed systems using conformance checking, a methodology borrowed from process mining. Distributed traces produced by instrumented applications are evaluated for conformance by comparison to design traces. The resulting conformance percentage is a quantitative metric that can be tracked over time to determine how closely a concrete implementation corresponds to the key attributes of the expected design model. This analysis is done using the dominant industry standard, OpenTelemetry, and so should apply to a wide range of distributed systems.