Benchmarking Quantum Computers via Protocols, Comparing IBM's Heron vs IBM's Eagle

TL;DR

This paper evaluates and compares the performance of IBM's Eagle and Heron quantum processors using a protocol-based benchmarking method that assesses practical quantum advantage at the protocol level.

Contribution

It extends a protocol-based benchmarking methodology to compare different quantum architectures, providing transparent insights into their operational strengths and limitations.

Findings

Heron architecture shows substantial performance improvements over Eagle.

The benchmarking approach offers a clear assessment of quantum processors' practical capabilities.

The study demonstrates the utility of protocol-level evaluation for quantum hardware comparison.

Abstract

As quantum computing hardware rapidly advances, objectively evaluating the capabilities and error rates of new processors remains a critical challenge for the field. A clear and realistic understanding of current quantum performance is essential to guide research priorities and drive meaningful progress. In this work, we apply and extend a protocol-based benchmarking methodology (Meirom, Mor, Weinstein Arxiv 2505.12441) that utilizes well-defined quantumness thresholds. By evaluating performance at protocol level rather then the gate level, this approach provides a transparent and intuitive assessment of whether specific quantum processors, or isolated sub-chips within them, can demonstrate a practical quantum advantage. To illustrate the utility of this method, we compare two generations of IBM quantum computers: the older Eagle architecture and the newer Heron architecture. Our findings reveal the genuine operational strengths and limitations of these devices, demonstrating substantial performance improvements in the newer Heron generation. This work was made possible by IBM Quantum policies which enable independent and objective assessment on their quantum computers and sub-chips. We strongly encourage other companies to emulate the independent qubit availability and the fair pricing which allow researchers to perform such assessments.