QASMBench: A Low-level QASM Benchmark Suite for NISQ Evaluation and Simulation

TL;DR

QASMBench is a comprehensive low-level benchmark suite based on OpenQASM, designed to evaluate NISQ devices, simulators, and compilers through diverse quantum kernels and novel metrics, facilitating performance insights and optimization.

Contribution

This work introduces QASMBench, a versatile benchmark suite with new metrics for analyzing NISQ device performance, error susceptibility, and optimization potential across multiple quantum platforms.

Findings

QASMBench effectively characterizes NISQ device performance.

Fidelity varies significantly among different quantum hardware.

New metrics provide deeper insights into quantum circuit execution.

Abstract

The rapid development of quantum computing (QC) in the NISQ era urgently demands a low-level benchmark suite and insightful evaluation metrics for characterizing the properties of prototype NISQ devices, the efficiency of QC programming compilers, schedulers and assemblers, and the capability of quantum system simulators in a classical computer. In this work, we fill this gap by proposing a low-level, easy-to-use benchmark suite called QASMBench based on the OpenQASM assembly representation. It consolidates commonly used quantum routines and kernels from a variety of domains including chemistry, simulation, linear algebra, searching, optimization, arithmetic, machine learning, fault tolerance, cryptography, etc., trading-off between generality and usability. To analyze these kernels in terms of NISQ device execution, in addition to circuit width and depth, we propose four circuit metrics including gate density, retention lifespan, measurement density, and entanglement variance, to extract more insights about the execution efficiency, the susceptibility to NISQ error, and the potential gain from machine-specific optimizations. Applications in QASMBench can be launched and verified on several NISQ platforms, including IBM-Q, Rigetti, IonQ and Quantinuum. For evaluation, we measure the execution fidelity of a subset of QASMBench applications on 12 IBM-Q machines through density matrix state tomography, which comprises 25K circuit evaluations. We also compare the fidelity of executions among the IBM-Q machines, the IonQ QPU and the Rigetti Aspen M-1 system. QASMBench is released at: http://github.com/pnnl/QASMBench.