Metriq: A Collaborative Platform for Benchmarking Quantum Computers

TL;DR

Metriq is an open-source platform that standardizes and facilitates reproducible benchmarking of quantum computers across different hardware, enabling systematic comparison and ongoing refinement of quantum performance metrics.

Contribution

It introduces a unified workflow for quantum benchmarking, including a comprehensive benchmark suite, a composite score, and a shared dataset for cross-platform evaluation.

Findings

Collected data from over ten quantum computers across vendors

Revealed strengths and limitations of different quantum benchmarks

Established correlations between benchmarks and hardware calibration metrics

Abstract

The fragmented landscape of quantum computer benchmarks, characterized by system-specific tools and inconsistent evaluation methodologies, hinders reliable cross-platform performance assessment. We introduce Metriq, an open-source collaborative platform for reproducible cross-platform quantum benchmarking that integrates benchmark definition and execution, data collection, and public presentation into a unified workflow. The Metriq benchmark suite spans both system-level metrics that characterize fundamental device properties such as entanglement quality, gate performance, and circuit speed, as well as application-inspired protocols that assess performance on quantum machine learning, optimization, and quantum simulation tasks. Benchmarks are chosen to scale with processor size, and the framework incorporates cost and resource estimation to support practical evaluation. Using Metriq, we collect and publicly release results from more than ten quantum computers across multiple hardware vendors, enabling systematic cross-platform comparison. The resulting curated dataset also reveals the practical strengths and limitations of individual benchmarks, creating a feedback loop that informs the ongoing refinement of the suite. To summarize performance across the benchmark suite, we introduce the Metriq Score, a composite index aggregating benchmark outcomes. We further present cross-benchmark analyses enabled by the shared dataset and their correlations with hardware calibration metrics. Through open development and data sharing, Metriq provides a practical foundation for reproducible benchmarking of quantum computers as hardware and benchmarking methods continue to evolve.