Systematic benchmarking of quantum computers: status and recommendations

TL;DR

This paper reviews the current state of quantum computer benchmarking, emphasizing the importance of standardized, multi-level evaluation methods to assess hardware, software, and application performance for scalable quantum computing.

Contribution

It provides a comprehensive overview of benchmarking techniques across various levels and offers recommendations for standardization and future development in quantum benchmarking.

Findings

Identifies key benchmarking levels: component, system, software, HPC, and application.

Highlights the need for standardization to ensure reproducibility.

Discusses ongoing efforts and future directions for quantum benchmarking.

Abstract

Architectures for quantum computing can only be scaled up when they are accompanied by suitable benchmarking techniques. The document provides a comprehensive overview of the state and recommendations for systematic benchmarking of quantum computers. Benchmarking is crucial for assessing the performance of quantum computers, including the hardware, software, as well as algorithms and applications. The document highlights key aspects such as component-level, system-level, software-level, HPC-level, and application-level benchmarks. Component-level benchmarks focus on the performance of individual qubits and gates, while system-level benchmarks evaluate the entire quantum processor. Software-level benchmarks consider the compiler's efficiency and error mitigation techniques. HPC-level and cloud benchmarks address integration with classical systems and cloud platforms, respectively. Application-level benchmarks measure performance in real-world use cases. The document also discusses the importance of standardization to ensure reproducibility and comparability of benchmarks, and highlights ongoing efforts in the quantum computing community towards establishing these benchmarks. Recommendations for future steps emphasize the need for developing standardized evaluation routines and integrating benchmarks with broader quantum technology activities.