Benchmarking Quantum Computers: Towards a Standard Performance Evaluation Approach

TL;DR

This paper reviews classical and quantum benchmarking methods, analyzes their limitations, and proposes guidelines to standardize quantum processor performance evaluation, aiming to establish a universal benchmarking organization.

Contribution

It introduces a comprehensive analysis of quantum benchmarking challenges and offers general guidelines for developing standardized performance evaluation methods for quantum computers.

Findings

Analysis of classical benchmarking attributes

Assessment of quantum benchmarking metrics

Proposed guidelines for quantum performance evaluation

Abstract

The technological development of increasingly larger quantum processors on different quantum platforms raises the problem of how to fairly compare their performance, known as quantum benchmarking of quantum processors. This is a challenge that computer scientists have already faced when comparing classical processors, leading to the development of various mathematical tools to address it, but also to the identification of the limits of this problem. In this work, we briefly review the most important aspects of both classical processor benchmarks and the metrics comprising them, providing precise definitions and analyzing the quality attributes that they should exhibit. Subsequently, we analyze the intrinsic properties that characterize the paradigm of quantum computing and hinder the naive transfer of strategies from classical benchmarking. However, we can still leverage some of the lessons learned such as the quality attributes of a \textit{good} benchmark. Additionally, we review some of the most important metrics and benchmarks for quantum processors proposed in the literature, assessing what quality attributes they fulfill. Finally, we propose general guidelines for quantum benchmarking. These guidelines aim to pave the way for establishing a roadmap towards standardizing the performance evaluation of quantum devices, ultimately leading to the creation of an organization akin to the Standard Performance Evaluation Corporation (SPEC).