Duet instrumentation: An Agentic Approach to Improving Sensitivity in Cloud Service Benchmarking

TL;DR

This paper introduces duet instrumentation, a novel LLM-enabled benchmarking approach that analyzes code changes to detect performance regressions more sensitively in cloud applications.

Contribution

It proposes a new LLM-based method for automated, targeted performance assessment of code changes, improving sensitivity over traditional benchmarks.

Findings

Achieves 58% precision, 93% recall, 71% specificity in identifying code changes.

Detects performance regressions at up to 5x lower severity than traditional methods.

Maintains similar latency distributions while improving regression detection sensitivity.

Abstract

Continuous cloud service performance benchmarking is essential for detecting performance bugs early before deploying them to production. However, detecting performance regressions using application benchmarks, which usually treat the system under test as a black box, is challenging due to variable I/O calls or changing performance characteristics of the underlying cloud infrastructure. Microbenchmarks are often more sensitive and accurate, but also more time-consuming to implement and run. Further, they do not capture the performance of the integrated system as a whole. A comprehensive performance assessment therefore typically requires a combination of both approaches. To address the shortcomings of application benchmarks, we propose duet instrumentation, a novel benchmarking paradigm enabled by recent advancements in large language model (LLM) code understanding. The idea is to analyze code changes between two consecutive application versions and measure performance differences directly at performance-relevant changes during a synchronized benchmark of both application versions, uncovering performance changes with higher sensitivity. We design a system that reliably automates the assessment and instrumentation of performance-relevant code changes between the two application versions. In experiments with a realistic testbed application offering configurable performance regressions, we find that our prototype achieves 58% precision, 93% recall, and 71% specificity (averaged across tasks) when comparing the generated instrumentation against the ideal instrumentation with a line-distance threshold of five. In the downstream application benchmark, we find that our prototype can detect performance regressions at up to 5x lower injected severity compared to a traditional duet application benchmark while preserving similar A/A latency distributions.