How to benchmark: the Measure-Explain-Test-Improve loop

TL;DR

This paper proposes a comprehensive methodology for conducting effective performance benchmarks in computer science research, especially in programming language studies, to improve evaluation practices.

Contribution

It introduces a structured Measure-Explain-Test-Improve loop to guide researchers in building robust performance benchmarks.

Findings

Provides practical recommendations for benchmarking in programming language research.

Clarifies the justification for each step in the benchmarking process.

Aims to help researchers develop solid performance evaluation methods.

Abstract

I would like to share recommendations on how to do performance benchmarks for the purpose of computer science research evaluation. Research in my field (programming language research) often involves performance considerations, but it is typically not the main tool used to evaluate our research (typically we evaluate via formal statements and their proofs, experience writing large or interesting examples, or systematic comparison of expressivity, feature set, etc.). My impression is that, as a result, we tend to not do our performance evaluation very well. In the present document I will try to explain a methodology to do benchmarking correctly (I hope!). People with no former benchmarking experience should be able to build solid performance evaluation as part of their research. I explain the justification for each aspect along the way.