Application-Motivated, Holistic Benchmarking of a Full Quantum Computing Stack

TL;DR

This paper introduces application-motivated benchmarking circuits for quantum computers, evaluating system performance based on noise, connectivity, and compilation strategies using IBM Quantum devices.

Contribution

It proposes a holistic benchmarking framework with three circuit classes tailored for practical quantum computing tasks, incorporating figures of merit requiring exponential classical resources.

Findings

Noise-aware compilation improves performance

Device connectivity impacts circuit execution success

Performance varies significantly with hardware noise levels

Abstract

Quantum computing systems need to be benchmarked in terms of practical tasks they would be expected to do. Here, we propose 3 "application-motivated" circuit classes for benchmarking: deep (relevant for state preparation in the variational quantum eigensolver algorithm), shallow (inspired by IQP-type circuits that might be useful for near-term quantum machine learning), and square (inspired by the quantum volume benchmark). We quantify the performance of a quantum computing system in running circuits from these classes using several figures of merit, all of which require exponential classical computing resources and a polynomial number of classical samples (bitstrings) from the system. We study how performance varies with the compilation strategy used and the device on which the circuit is run. Using systems made available by IBM Quantum, we examine their performance, showing that noise-aware compilation strategies may be beneficial, and that device connectivity and noise levels play a crucial role in the performance of the system according to our benchmarks.