Robust benchmarking in noisy environments

TL;DR

This paper introduces a robust benchmarking strategy that accounts for environmental noise and fluctuations, ensuring reliable performance measurements in noisy computing environments.

Contribution

It presents a new benchmarking approach resilient to environmental noise, supported by a model explaining nonideal timing statistics, and implements it in the BenchmarkTools Julia package.

Findings

Effective in noisy environments with timer errors and OS jitter

Validated through integration in Julia's CI pipelines

Provides a theoretical model for nonideal timing statistics

Abstract

We propose a benchmarking strategy that is robust in the presence of timer error, OS jitter and other environmental fluctuations, and is insensitive to the highly nonideal statistics produced by timing measurements. We construct a model that explains how these strongly nonideal statistics can arise from environmental fluctuations, and also justifies our proposed strategy. We implement this strategy in the BenchmarkTools Julia package, where it is used in production continuous integration (CI) pipelines for developing the Julia language and its ecosystem.