MapReplay: Trace-Driven Benchmark Generation for Java HashMap

TL;DR

MapReplay is a benchmarking method that creates realistic, efficient HashMap performance tests by replaying operation sequences and internal states, bridging the gap between microbenchmarks and full application benchmarks.

Contribution

It introduces MapReplay, a novel approach that combines realism and efficiency for HashMap benchmarking by replaying operation sequences and internal states.

Findings

Reproduces application-level performance trends accurately.

Reduces benchmarking time significantly.

Reveals insights difficult to observe with full benchmarks.

Abstract

Hash-based maps, particularly java.util.HashMap, are pervasive in Java applications and the JVM, making their performance critical. Evaluating optimizations is challenging because performance depends on factors such as operation patterns, key distributions, and resizing behavior. Microbenchmarks are fast and repeatable but often oversimplify workloads, failing to capture the realistic usage patterns. Application benchmarks (e.g., DaCapo, Renaissance) provide realistic usages but are more expensive to run, prone to variability, and dominated by non-HashMap computations, making map-related performance changes difficult to observe. To address this challenge, we propose MapReplay, a benchmarking methodology that combines the realism of application benchmarks with the efficiency of microbenchmarks. MapReplay traces HashMap API usages generating a replay workload that reproduces the same operation sequence while faithfully reconstructing internal map states. This enables realistic and efficient evaluation of alternative implementations under realistic usage patterns. Applying MapReplay to DaCapo-Chopin and Renaissance, the resulting suite, MapReplayBench, reproduces application-level performance trends while reducing experimentation time and revealing insights difficult to obtain from full benchmarks.