Investigating the Impact of Isolation on Synchronized Benchmarks

TL;DR

This paper evaluates three isolation strategies—cgroups and CPU pinning, Docker containers, and Firecracker MicroVMs—to improve benchmarking accuracy in cloud environments by reducing performance variability caused by resource contention.

Contribution

It provides a comparative analysis of different isolation techniques for synchronized workloads, highlighting their effectiveness in mitigating noise during benchmarking.

Findings

Process isolation reduces false positives in benchmarks.

Docker containers are more susceptible to noise than other methods.

Firecracker MicroVMs offer effective isolation for synchronized workloads.

Abstract

Benchmarking in cloud environments suffers from performance variability from multi-tenant resource contention. Duet benchmarking mitigates this by running two workload versions concurrently on the same VM, exposing them to identical external interference. However, intra-VM contention between synchronized workloads necessitates additional isolation mechanisms. This work evaluates three such strategies: cgroups and CPU pinning, Docker containers, and Firecracker MicroVMs. We compare all strategies with an unisolated baseline experiment, by running benchmarks with a duet setup alongside a noise generator. This noise generator "steals" compute resources to degrade performance measurements. All experiments showed different latency distributions while under the effects of noise generation, but results show that process isolation generally lowered false positives, except for our experiments with Docker containers. Even though Docker containers rely internally on cgroups and CPU pinning, they were more susceptible to performance degradation due to noise influence. Therefore, we recommend to use process isolation for synchronized workloads, with the exception of Docker containers.