Fair Benchmarking of Optimisation Applications

TL;DR

This paper proposes a comprehensive framework for fairly benchmarking quantum optimisation methods, addressing unique challenges like continuous dynamics and probabilistic outcomes to ensure trustworthy and reproducible performance assessments.

Contribution

It introduces principles and protocols for fair benchmarking of quantum optimisation, incorporating application-driven metrics and emphasizing transparency and reproducibility.

Findings

Framework enables responsible evaluation of quantum methods

Incorporates energy-aware and application-driven metrics

Ensures reproducibility and comparability of results

Abstract

Quantum optimisation is emerging as a promising approach alongside classical heuristics and specialised hardware, yet its performance is often difficult to assess fairly. Traditional benchmarking methods, rooted in digital complexity theory, do not directly capture the continuous dynamics, probabilistic outcomes, and workflow overheads of quantum and hybrid systems. This paper proposes principles and protocols for fair benchmarking of quantum optimisation, emphasising end-to-end workflows, transparency in tuning and reporting, problem diversity, and avoidance of speculative claims. By extending lessons from classical benchmarking and incorporating application-driven and energy-aware metrics, we outline a framework that enables practitioners to evaluate quantum methods responsibly, ensuring reproducibility, comparability, and trust in reported results.