Empirical Algorithmics: draw your own conclusions

TL;DR

This paper critically examines the practice of rescaling published algorithm performance results in empirical studies, highlighting potential pitfalls and the risk of misleading conclusions in algorithm comparisons.

Contribution

It reveals the dangers of rescaling techniques in empirical algorithmics and emphasizes the need for careful interpretation of such comparative results.

Findings

Rescaling can lead to conflicting conclusions.

Unsafe rescaling may misrepresent algorithm performance.

Empirical comparisons require cautious analysis.

Abstract

In an empirical comparisons of algorithms we might compare run times over a set of benchmark problems to decide which one is fastest, i.e. an algorithmic horse race. Ideally we would like to download source code for the algorithms, compile and then run on our machine. Sometimes code isn't available to download and sometimes resource isn't available to implement all the algorithms we want to study. To get round this, published results are rescaled, a technique endorsed by DIMACS, and those rescaled results included in a new study. This technique is frequently used when presenting new algorithms for the maximum clique problem. We demonstrate that this is unsafe, and that if carelessly used may allow us to draw conflicting conclusions from our empirical study.