Practical Insights into Fair Comparison and Evaluation Frame for Neutral-Atom Compilers

TL;DR

This paper introduces a unified evaluation framework and representation for neutral-atom quantum compilers, enabling fair comparison and revealing that previous performance gaps are often overstated.

Contribution

The authors present RSQASM, a standardized post-compilation representation, and provide tools to enable consistent evaluation of neutral-atom compiler performance.

Findings

Re-evaluation shows performance gaps are smaller than previously reported.

The framework reveals that some claimed advantages of certain compilers are not statistically significant.

Inconsistent evaluation metrics in prior work can lead to misleading conclusions.

Abstract

Neutral-atom quantum computing is among the most promising platforms for scalable quantum computation, and compilation toolchains are crucial for leveraging capabilities such as qubit shuttling and parallel gate execution. An important challenge, however, is that existing neutral-atom compilers are often evaluated using metrics computed over different parts of the toolchain and under non-equivalent assumptions. Consequently, fair quantification and comparison of compiler performance remain difficult. Reported metrics may depend on inconsistent transpilation optimization levels, different movement-duration models, different sets of considered fidelity sources, and even minor implementation bugs or undocumented representation choices. To address this problem, we present a unified and reproducible evaluation framework for neutral-atom compilers. Our framework introduces RSQASM (Routed and Scheduled QASM), a QASM-inspired post-compilation representation that captures mapped, routed, and scheduled circuits, including explicit parallel gate execution and shuttling operations. As part of the framework, we provide adapter scripts that translate existing compiler outputs and intermediate artifacts into RSQASM. As a case study, we compare three well-known neutral-atom compilation toolchains: HybridMapper, DasAtom, and Enola, motivated by the large performance differences reported in prior work. Using our framework and representation, we perform a new evaluation and show that several previously claimed performance gaps become substantially smaller and, in some cases, are not reproduced once evaluation inconsistencies are removed.