Mint: Cost-Efficient Tracing with All Requests Collection via Commonality and Variability Analysis

TL;DR

Mint introduces a novel 'commonality + variability' approach for distributed trace collection, capturing all requests efficiently while significantly reducing storage and network costs, outperforming traditional sampling methods.

Contribution

The paper proposes a cost-efficient tracing framework, Mint, that shifts from binary sampling to pattern-based trace aggregation for improved information retention and resource optimization.

Findings

Mint captures all traces with minimal storage overhead.

Trace storage reduced to an average of 2.7%.

Network overhead reduced to an average of 4.2%.

Abstract

Distributed traces contain valuable information but are often massive in volume, posing a core challenge in tracing framework design: balancing the tradeoff between preserving essential trace information and reducing trace volume. To address this tradeoff, previous approaches typically used a '1 or 0' sampling strategy: retaining sampled traces while completely discarding unsampled ones. However, based on an empirical study on real-world production traces, we discover that the '1 or 0' strategy actually fails to effectively balance this tradeoff. To achieve a more balanced outcome, we shift the strategy from the '1 or 0' paradigm to the 'commonality + variability' paradigm. The core of 'commonality + variability' paradigm is to first parse traces into common patterns and variable parameters, then aggregate the patterns and filter the parameters. We propose a cost-efficient tracing framework, Mint, which implements the 'commonality + variability' paradigm on the agent side to enable all requests capturing. Our experiments show that Mint can capture all traces and retain more trace information while optimizing trace storage (reduced to an average of 2.7%) and network overhead (reduced to an average of 4.2%). Moreover, experiments also demonstrate that Mint is lightweight enough for production use.