Benchmarking the Operation of Quantum Heuristics and Ising Machines: Scoring Parameter Setting Strategies on Optimization Applications

TL;DR

This paper proposes a benchmarking framework for quantum heuristics and Ising machines, emphasizing parameter setting strategies and their impact on optimization performance, supported by an open-source evaluation tool.

Contribution

It introduces a standardized benchmarking procedure and an open-source software package for evaluating parameter tuning strategies in quantum and Ising-based optimization solvers.

Findings

Parameter setting significantly affects solver performance.

The benchmarking method provides statistical performance analysis.

Open-source tool aids in designing and visualizing tuning strategies.

Abstract

We discuss guidelines for evaluating the performance of parameterized stochastic solvers for optimization problems, with particular attention to systems that employ novel hardware, such as digital quantum processors running variational algorithms, analog processors performing quantum annealing, or coherent Ising Machines. We illustrate through an example a benchmarking procedure grounded in the statistical analysis of the expectation of a given performance metric measured in a test environment. In particular, we discuss the necessity and cost of setting parameters that affect the algorithm's performance. The optimal value of these parameters could vary significantly between instances of the same target problem. We present an open-source software package that facilitates the design, evaluation, and visualization of practical parameter tuning strategies for complex use of the heterogeneous components of the solver. We examine in detail an example using parallel tempering and a simulator of a photonic Coherent Ising Machine computing and display the scoring of an illustrative baseline family of parameter-setting strategies that feature an exploration-exploitation trade-off.