Machines are benchmarked by code, not algorithms

TL;DR

This paper emphasizes the importance of benchmarking based on actual machine code rather than source code, as small modifications can significantly affect performance measurements on hardware.

Contribution

It advocates for reproducibility in machine benchmarking by focusing on the machine code executed, not just the source code or compilation options.

Findings

Small code modifications impact benchmark results

Reproducibility requires analyzing actual machine code

Benchmarking should focus on executable code for accuracy

Abstract

This article highlights how small modifications to either the source code of a benchmark program or the compilation options may impact its behavior on a specific machine. It argues that for evaluating machines, benchmark providers and users be careful to ensure reproducibility of results based on the machine code actually running on the hardware and not just source code. The article uses color to grayscale conversion of digital images as a running example.