Statistical Assertions for Validating Patterns and Finding Bugs in Quantum Programs

TL;DR

This paper introduces statistical assertions for quantum programs, enabling debugging by checking if quantum states match expected classical or quantum properties, thus aiding development and bug detection in quantum computing.

Contribution

It presents a novel approach to debugging quantum programs using statistical assertions integrated into a quantum programming language.

Findings

Effective bug detection in quantum algorithms

Identification of common quantum bugs

Strategies for placing assertions to prevent bugs

Abstract

In support of the growing interest in quantum computing experimentation, programmers need new tools to write quantum algorithms as program code. Compared to debugging classical programs, debugging quantum programs is difficult because programmers have limited ability to probe the internal states of quantum programs; those states are difficult to interpret even when observations exist; and programmers do not yet have guidelines for what to check for when building quantum programs. In this work, we present quantum program assertions based on statistical tests on classical observations. These allow programmers to decide if a quantum program state matches its expected value in one of classical, superposition, or entangled types of states. We extend an existing quantum programming language with the ability to specify quantum assertions, which our tool then checks in a quantum program simulator. We use these assertions to debug three benchmark quantum programs in factoring, search, and chemistry. We share what types of bugs are possible, and lay out a strategy for using quantum programming patterns to place assertions and prevent bugs.

Tables

Table 1. Table 1. Correct and incorrect code for rotation decomposition. Using the Scaffold language  (JavadiAbhari et al . , 2014 ) as an example, we code out Figure  3 ’s controlled operation U, where U is a rotation in just one axis. Because only one axis is needed, we can drop either operation A or C, paying attention to the sign on the angles. Reordering the lines of code or signs results in a rotation in the wrong direction.

Correct, operation A unneeded	Correct, operation C unneeded	Incorrect, angles flipped
Rz(q1,+angle/2); // C	CNOT(q0,q1);	Rz(q1,-angle/2);
CNOT(q0,q1);	Rz(q1,-angle/2); // B	CNOT(q0,q1);
Rz(q1,-angle/2); // B	CNOT(q0,q1);	Rz(q1,+angle/2);
CNOT(q0,q1);	Rz(q1,+angle/2); // A	CNOT(q0,q1);
Rz(q0,+angle/2); // D	Rz(q0,+angle/2); // D	Rz(q0,+angle/2); // D

Table 2. Table 2. Correct classical input a 𝑎 a and a − 1 superscript 𝑎 1 a^{-1} to Shor’s algorithm for factoring 15, using 7 as a guess.

$k$, the algorithm iteration	0	1	2	3	…
$a=7^{2^k}mod15$	7	4	1	1	…
$a^{-1}$; $a\times a^{-1}\equiv 1mod15$	13	4	1	1	…

Table 3. Table 3. Probability of measuring values of outputs and ancillary qubits of Shor’s algorithm, with incorrect inputs ( a − 1 = 12 superscript 𝑎 1 12 a^{-1}=12 instead of 13 13 13 on first iteration). If the ancillary qubits collapse to zero on measurement, the algorithm still succeeds, returning correct outputs of 0 0 , 2 2 2 , 4 4 4 , 6 6 6   (Nielsen and Chuang, 2011 , p. 235) . However, the possibility of measuring non-zero for the ancillary qubits indicates a bug.

Probability		Output measurement
		0	1	2	3	4	5	6	7
Ancillary	0	1/8	0	1/8	0	1/8	0	1/8	0
qubits	2	1/64	1/64	1/64	1/64	1/64	1/64	1/64	1/64
measurement	7	1/64	1/64	1/64	1/64	1/64	1/64	1/64	1/64
	8	1/64	1/64	1/64	1/64	1/64	1/64	1/64	1/64
	13	1/64	1/64	1/64	1/64	1/64	1/64	1/64	1/64

Table 4. Table 4. Grover’s amplitude amplification subroutine in two languages, showcasing QC-specific language syntax for reversible computation (rows 2 & 6) and controlled operations (rows 3 & 5), exposing structure that can guide placing assertions.

	Scaffold (C syntax) (JavadiAbhari et al., 2014)	ProjectQ (Python syntax) (Steiger et al., 2018)
1	int j; qbit ancilla[n-1]; // scratch register for(j=0; j<n-1; j++) PrepZ(ancilla[j],0);	# reflection across # uniform superposition
2	// Hadamard on q for(j=0; j<n; j++) H(q[j]); // Phase flip on q = 0...0 so invert q for(j=0; j<n; j++) X(q[j]);	with Compute(eng):       All(H) | q       All(X) | q
3	// Compute x[n-2] = q[0] and ... and q[n-1] CCNOT(q[1], q[0], ancilla[0]); for(j=1; j<n-1; j++)       CCNOT(ancilla[j-1], q[j+1], ancilla[j]);	with Control(eng, q[0:-1]):
4	// Phase flip Z if q=00...0 cZ(ancilla[n-2], q[n-1]);	Z | q[-1]
5	// Undo the local registers for(j=n-2; j>0; j--)       CCNOT(ancilla[j-1], q[j+1], ancilla[j]); CCNOT(q[1], q[0], ancilla[0]);	# ProjectQ automatically # uncomputes control
6	// Restore q for(j=0; j<n; j++) X(q[j]); for(j=0; j<n; j++) H(q[j]);	Uncompute(eng)

Table 5. Table 5. QC calculated energy for H 2 subscript H 2 \mathrm{H_{2}} (bond length = 73.48 pm) for different electron assignments.

	Electron assignments				QC calculated energy (relative)
	Bonding		Antibonding
	$\uparrow$	$\downarrow$	$\uparrow$	$\downarrow$
\nth3 excited state (E3)	0	0	1	1	-0.164
\nth2 excited state (E2)	0	1	1	0	-0.217
	1	0	0	1
\nth1 excited state (E1)	0	1	0	1	-0.244
	1	0	1	0
Ground state (G)	1	1	0	0	-0.295

Full text

quantum computing, correctness, program patterns, assertions, debugging, validation, chi-square test

††journalyear: 2019††copyright: acmcopyright††conference: The 46th Annual International Symposium on Computer Architecture; June 22–26, 2019; Phoenix, AZ, USA††booktitle: The 46th Annual International Symposium on Computer Architecture (ISCA ’19), June 22–26, 2019, Phoenix, AZ, USA††price: 15.00††doi: 10.1145/3307650.3322213††isbn: 978-1-4503-6669-4/19/06††ccs: Computer systems organization Quantum computing††ccs: Software and its engineering Software testing and debugging††ccs: Hardware Quantum computation††ccs: Software and its engineering Patterns††ccs: Theory of computation Assertions

1. Introduction

Quantum computing is reaching an inflection point (Preskill, 2018; National Academies of Sciences, Engineering, and Medicine, 2019). After years of work on low-level quantum computing (QC) devices, small but viable QC prototypes are now available to run programs. These QC prototypes are increasing in size, with much research attention being placed on improving their reliability and increasing the counts of qubits (quantum bits), the fundamental building block for QC (Ladd et al., 2010; Linke et al., 2017; Tannu and Qureshi, 2019; Murali et al., 2019). These advancements in QC hardware may soon lead to a demonstration of a quantum algorithm running on QC hardware that exceeds the performance of a classical computer system (Harrow and Montanaro, 2017; Boixo et al., 2018). Such a demonstration would move the world closer to a new era of computing where QC systems solve problems in chemistry (McArdle et al., 2018; Olson et al., 2017), optimization (Farhi et al., 2000; Farhi et al., 2014; Pakin, 2019), and even cryptography (Shor, 1997) that are currently intractable with classical computers.

With small-scale machines available to run real code, a natural challenge lies in bridging the QC architectural gap between algorithms and hardware. One aspect of that gap is in creating correct programs to run on quantum computers (Chong et al., 2017; Häner et al., 2018). Until recently, QC algorithms existed only in the form of abstract specifications and equations, and were rarely programmed for actual execution or simulation, and therefore relatively little QC debugging has ever occurred. Furthermore, QC debugging faces challenges beyond that of classical computing. In particular, typical debugging approaches based on printing out variable values during program execution do not easily apply to QC programs, because program states in QC collapse to classical values when observed. Second, while programmers have more freedom to observe full program states in QC simulations on classical computers, the massive state spaces of QC executions limits this approach to small programs. Finally, even when limited simulations are tractable, it can be difficult to interpret the simulation results.

While the problem of debugging and validating quantum programs has been extensively identified as a major barrier to useful quantum computation (Chong et al., 2017; Häner et al., 2018; Green et al., 2013; Svore and Troyer, 2016; Roetteler et al., 2017; Paykin et al., 2017; National Academies of Sciences, Engineering, and Medicine, 2019), little has been said about what actually constitutes a quantum program bug. Similarly limited detail has been shared about the inside story of translating QC algorithms in to working QC programs, even though the field is now making rapid progress in writing open source QC program benchmarks across several quantum programming languages (LaRose, 2019; Green et al., 2013; JavadiAbhari et al., 2014; Steiger et al., 2018; Svore et al., 2018; McClean et al., 2017).

This work shares the detailed process of debugging quantum programs, with the help of a proposed set of quantum program assertions and breakpoints based on statistical tests. Using ensembles of classical observations taken from the intermediate state of quantum programs, these statistical tests are able to decide if the program state belongs to one of three important classes of quantum states: classical, superposition, and entangled. With this information, programmers can determine if the execution of the quantum program is valid up to each breakpoint. If the program state is invalid, the assertions guide the programmer in finding the bug inside subroutines and in program code up to that point.

For three quantum program benchmarks in factoring integers, database search, and quantum chemistry, we describe what kinds of bugs occurred in the process of bringing up the programs from unit tests to integration testing. We categorize the bugs according to where in the structure of quantum programs they may arise, and we lay out a strategy for placing statistical assertions that effectively catches them.

The rest of this paper is organized as follows: Section 2 provides relevant background for quantum programs and debugging. Section 3 details our statistical assertions and simulation framework for debugging quantum programs, which is then used in Section 4 for building and debugging an integer factorization quantum program. Section 5 evaluates the use of the assertions in two additional case studies. Section 6 discusses related approaches to writing correct quantum programs.

2. Background on quantum states and quantum programs

First, we review the principles of quantum computing (Kaye et al., 2007; Mermin, 2007; Metodi et al., 2011; Nielsen and Chuang, 2011), in order to understand how debugging quantum programs is different from and more challenging than classical debugging.

2.1. Qubits, superpositions, and entanglement

The basic unit of information in QC is the qubit, which can take on values of $\ket{0}$ and $\ket{1}$ like bits in classical computing, but unlike classical bits, qubits can also be in a probabilistic superposition between the two values. Quantum computers can also measure the value of a qubit, forcing it to collapse out of superposition into a classical value such as ‘0’ or ‘1’. Measurement disturbs the values of variables in a quantum computer. So unlike the case in classical computing, programmers cannot easily pause execution and observe the values of qubits as a quantum program runs. As a result of this limited visibility into program state, programmers must carefully choose what to measure and test when they debug quantum programs.

Aside from qubits being in superposition states, the other feature of data in QC is entanglement. For example, when the states of two qubits are entangled, the combined state of the two-qubit system must be viewed as a superposition of a larger set of elementary states $\ket{00}$, $\ket{01}$, $\ket{10}$, and $\ket{11}$. An entangled state cannot be factored into independent pieces of information, and can no longer be viewed as a simple concatenation of two qubits. Likewise, a three-qubit system has potential superpositions of eight states, and so on. For this reason, as more qubits come into play in a quantum computer, the number of states that data can be in grows exponentially. This exponential growth of possible values due to superpositions and entanglement underlies the power of QC.

On the flip side, the huge state spaces involved in QC limits programmers’ ability to use classical computers to simulate and debug QC programs. Naïve simulation of a 50-qubit quantum computer, for example, needs $2^{50}$ or roughly one quadrillion floating point numbers just to store the program state at any instant (Häner and Steiger, 2017). While more advanced techniques can decrease the memory requirement for simulating circuits (Zulehner and Wille, 2019; Miller et al., 2006; Markov and Shi, 2008; Viamontes et al., 2009; Wu et al., 2018), interactive programming and simulating quantum programs on a workstation is still limited to 20 to 30 qubits. For this reason, testing and debugging QC programs in simulation is only possible for toy-sized programs.

2.2. Quantum computer operations,

programs, and a taxonomy for bugs

The process of quantum computing involves applying operations on qubits. Quantum computer scientists use diagrams such as Figure 1 to represent sequences of quantum operations. Looking at Figure 1 one sees that quantum programs consist of three conceptual parts:

1. (1)

Inputs to quantum algorithms include quantum initial values for qubits and classical input parameters such as coefficients for rotations. Getting these inputs to be correct is the focus of Section 4.1. 2. (2)

Operations include the specification of how to create an entangled state shown in Figure 1. Getting these basic operations to be correct is the focus of Section 4.2. Additionally, operations can be further composed according to patterns such as iteration, recursion, and mirroring. The correctness of these code patterns is the focus of Sections 4.3, 4.4, and 4.5. 3. (3)

Outputs of quantum algorithms are the final classical measurement values of qubits such as $m_{0}$ and $m_{1}$. Furthermore, any temporary variables used in the course of a program have to be safely disentangled from the rest of the quantum state and discarded. Getting these final results to be correct is the focus of Section 4.6.

Bugs in quantum programs can crop up in any of these three parts of a QC program due to mistakes in converting algorithm specifications to program code. We will give examples of bugs in each of these places using detailed case studies of real quantum programs. To our knowledge, our work is the first study of such QC program patterns and anti-patterns (Huang and Martonosi, 2019).

3. Our approach to statistical quantum program assertions

Even though it is hard to have as much visibility into quantum program state as is the case in classical computing, limited but useful assertions checking is possible, particularly for the purposes of writing correct quantum programs. In this paper, we propose using statistical tests on measured outputs as a way to gain visibility into a quantum program (Section 3.1). We implement several quantum programs in a quantum programming language (Section 3.2). Using our “quantum breakpoints,” programmers are able to check for expected values at various points in simulated quantum program runs, allowing them to debug the programs with the aid of our statistical assertions (Section 3.3).

3.1. Quantum assertions using statistical tests

In this subsection, we preview three basic types of assertions useful in quantum debugging, and we discuss the mechanics of using statistical tests as quantum assertions.

Following our overview of quantum states, superpositions, and entanglement in Section 2, one already sees that there are three kinds of possible assertions in a quantum program:

1. (1)

Classical assertions: a quantum variable should take on a deterministic (classical) integer value upon measurement; 2. (2)

Superposition assertions: a quantum variable in superposition should take on a probabilistic distribution of multiple values upon measurement; 3. (3)

Entanglement assertions: two or more quantum variables in an entangled state should take on associated (correlated)111Correlation is dependence between variables whose magnitude does matter. That compares with association which is dependence between nominal variables that are merely categories, and whose magnitude don’t matter. values once they are measured.

Statistical tests use ensembles of multiple measurements to decide to reject hypotheses. These serve as quantum programming assertions. With enough measurements, a statistical test is able to tell that an assertion does not hold, indicating a bug in the program. Otherwise, if the assertion holds, programmers can proceed cautiously knowing the quantum state so far is consistent with “no bug,” given the number of measurements provided to the statistical test. While this approach is not powerful enough to decisively conclude that a quantum program state is correct, the debugging experience we share in this paper shows that detecting incorrect states is still useful enough to catch program bugs.

Specifically, our tool uses the chi-square test to check for classical and superposition quantum states, and it uses contingency table analysis coupled with the chi-square test to check for entangled states (Press et al., 2007). The assertions on classical and superposition quantum states are useful for quantum algorithm precondition checks and for unit testing, discussed in Sections 4.1, 4.2, 4.3, and 4.6. Similarly, the assertions on entangled states are useful for checking interaction between qubits, discussed in Sections 4.4 and 4.5.

3.2. Benchmark QC algorithms for debugging

To demonstrate using our assertions framework to debug quantum programs, we focus this paper on debugging three programs in factoring integers, database search, and quantum chemistry, each representing a different class of quantum algorithms.

Using the Shor’s integer factoring quantum algorithm for factoring integers as a centerpiece example throughout Section 4, we show how the structure of quantum programs guides programmers where to put useful quantum assertions. We propose a complete taxonomy of where bugs can take place, and show assertions can catch all of the categories of bugs.

Then, using the Grover’s database search and a quantum chemistry problem as additional case studies in Section 5, we discuss how different classes of quantum algorithms present different opportunities and challenges for debugging.

3.3. Simulation and assertions checking methodology

We implement the programs in the Scaffold quantum programming language developed by our research group (JavadiAbhari et al., 2014),222https://github.com/epiqc/ScaffCC now augmented with the ability to specify and check for assertions. The assertions instruct the compiler where to stop execution and measure qubit states. Since premature measurement destroys the quantum state, the assertions effectively terminate the quantum program, splitting the quantum program into multiple breakpoints.

Our tool uses the ScaffCC compiler to compile Scaffold code with assertions into multiple versions of OpenQASM, a QC assembly language (Cross et al., 2017). Each version of the compiled program has the program execution up to the quantum breakpoint, followed by an early measurement and assertions on expected values for the quantum variables.

Then, our tool simulates an ensemble of executions for each of the programs ending at each breakpoint, using the QX quantum simulator (Khammassi et al., 2017) running on a cluster compute infrastructure.

Finally, the tool gathers the results from the simulations to check for the assertions at each breakpoint. The measurement results feed into statistical tests, which check if the quantum variables have values that are consistent with being in one of classical, superposition, or entangled states. If the statistical tests reject the null hypotheses, that indicates the assertions were violated, which means there is a bug in the quantum program or that the assertion was an incorrect constraint.

Given the rapid growth of QC infrastructure, programmers now have the chance to test a variety of quantum algorithms written in many languages (LaRose, 2019). To validate our overall approach, we cross-validated our quantum programs and simulation results against equivalent programs written in other quantum programming languages, such as LIQUi—¿ (Roetteler et al., 2017), ProjectQ (Häner et al., 2016; Steiger et al., 2018), and Q# (Svore et al., 2018).

From this detailed debugging experience spanning multiple algorithms, languages, and simulators, we are able to concretely describe for the first time what types of bugs may crop up in quantum programs, and how assertions can aid in the debugging of quantum programs. As an added contribution, Section 5 discusses how language features of different QC programming languages can aid with the placement of quantum assertions, or otherwise prevent bugs in the first place.

4. QC debugging and assertions:

Shor’s algorithm case study

Using the Shor’s quantum algorithm for factoring integers as a concrete case study, we show how the structure of the quantum program shown in Figure 2 aids programmers in using assertions to debug quantum programs. To defend against bugs in quantum program input, operations, and outputs, programmers can write assertions that check for preconditions, invariants, and postconditions. These constraints aid in the process of bringing up the program from unit tests to overall integration tests.

Shor’s factorization algorithm uses a quantum computer to factor a composite number in polynomial time complexity, providing exponential speedup relative to the best known classical algorithms (Shor, 1997). The algorithm works by estimating eigenvalues of a matrix, where the matrix is generated from the exponentiation of an integer representing a trial factor. The arithmetic is done in modular space with the modulus $N$ set to be the integer one wants to factor. Here, we replicate results for factoring $N=15$, the simplest example (Lanyon et al., 2007) (Nielsen and Chuang, 2011, p. 235), by following an example for an implementation that minimizes the qubit cost (Beauregard, 2003). Once the quantum part of Shor’s algorithm is done finding the eigenvalues, those values are useful in a classical post-processing algorithm to find 3 and 5, the factors of 15.

We focus on debugging the Shor’s factoring algorithm because it features in a single overall algorithm several important primitives (kernels) and program patterns common to many quantum algorithms. The primitives invoked in Shor’s algorithm include order finding, eigenvalue estimation, state preparation, phase estimation, and quantum Fourier transform. Our assertions and debugging techniques apply to several other QC applications that invoke similar primitives and patterns.

4.1. Classical and superposition precondition assertions on quantum initial values

Correct implementation and execution of QC programs begins with the right input states. Given their importance, it is worthwhile to check these preconditions by running or simulating programs up to the entry point of subroutines, and performing a premature measurement to check for these anticipated states.

Bug type 1: Incorrect quantum initial values.

The type of quantum initial state that an algorithm needs depends on the type of algorithm. For eigenvalue estimation algorithms such as Shor’s algorithm, the two major pieces of quantum data are an upper register and a lower register (far left of Figure 2): the upper register participates in a phase estimation subroutine; while the lower register is scratch space to implement a mathematical function such as modular exponentiation.

Here, the lower target register needs to be initialized to a strictly classical integer value, such as ‘1’. That means that if a quantum computer measured the qubit encoding the least significant bit of the quantum variable, it should return ‘1’, while the measurements on the other qubits of the variable should return ‘0’.

On the other hand, the upper control register needs to be initialized to a uniform superposition of values. More concretely, if the upper register consists of, for example, 3 qubits, the measurement of the upper register at the beginning of the algorithm should return the eight values ‘000’, ‘001’, … ‘111’ with equal probability. That uniform superposition of values is created by the quantum Fourier transform (QFT). The QFT operation has the effect of taking integer inputs, and re-encoding them as quantum values that are distinct from each other by a quantum property known as phase.

The classical value in the upper register and uniform superposition in the lower register are the preconditions for Shor’s algorithm. Other types of preconditions are possible for other types of algorithms. For example, quantum communications protocols often need entangled states as initial conditions.

Defense type 1: Assertion checks for classical and superposition preconditions.

Our tool checks for both classical and superposition states using chi-square statistical tests on measured values.

To test for classical integer values, our tool gives the chi-square test the hypothesis that the distribution is unimodal with a peak at the expected value. If the test returns a small $p$-value ($\leq 0.05$), then the null hypothesis is rejected, meaning the initial state cannot be the expected value, indicating a violation of the precondition. If, on the other hand, the test returns a large $p$-value, typically close to 1.0, then the input state is consistent with being the expected value, though programmers cannot completely rule out a precondition violation. If there actually was a bug, programmers would only be able to detect the precondition violation using more measurements.

To test for superposition quantum states of $n$-qubits, the chi-square test uses as its hypothesis that the measurements should be a uniform distribution across all $2^{n}$ integer values. If the superposition precondition is violated, and there are sufficient measurements, the values would be concentrated enough for the chi-square test to reject the null hypothesis and raise an exception.

In prior work, the Q# quantum programming language has support for assertion checks for integer values, and is able to check for such assertions in simulations of quantum programs (Svore et al., 2018). To our knowledge, this paper is the first proposal for quantum assertions on superposition and (later in this paper) entangled quantum states. Furthermore, this is the first work to discuss using statistical tests to check for these hypotheses.

4.2. Unit tests for a library of subroutines

Now that we have made sure the quantum initial states are valid, the next step in programming the Shor’s algorithm is to build up the algorithm operations. We do so starting from elementary operations, which we exhaustively validate against their closed form solutions, and against implementations in other languages. Then we compose the elementary operations following code patterns—iterations, recursion, and mirroring—and test the composite subroutines using assertions (Sections 4.3, 4.4, 4.5).

Bug type 2: Incorrect operations and transformations.

In order to correctly implement Shor’s algorithm, programmers first have to build up the quantum subroutines such as the controlled rotation subroutine depicted in Figure 3. This subroutine is the building block for QFT and adder routines in Shor’s algorithm (modules in Figure 2). Typically this task consists of translating quantum circuit diagrams, such as Figure 3, into quantum program code. Sometimes, programmers do not even have quantum circuit diagrams and must instead start with equation descriptions for the operations they need. This process of converting specifications to program code is unintuitive and tricky. For example, Table 1 lists multiple ways to code the decomposition of the controlled rotation, including a buggy one where a small mistake leads to the wrong operation.

Defense type 2: Assertion checks for unit testing.

An obvious defense against coding mistakes in basic subroutines (such as controlled rotation, QFT, and addition subroutines) is to use a library of shared code. Doing so helps ensure program correctness by allowing programmers to exhaustively validate small subroutines, in order to bootstrap larger subroutines. Unit testing is especially important in QC as running or simulating large quantum programs is costly, making larger scale integration tests impossible.

As a concrete example, we use precondition and postcondition assertion checks inside a test harness to validate the QFT subroutine, another important building block. As shown in Listing 1, first the program prepares a classical integer state (Lines 5-9). Then, the program checks as a precondition of the QFT subroutine that the input is a classical integer value, in this case ‘5’ (Line 12). The corresponding postcondition of the QFT subroutine is that the output should be a uniform superposition if the program collapses the quantum state and measure the values at that point (Line 18).

While these simple constraints are not enough on their own to validate that the QFT implementation and its sub-components are correct, they are valuable lightweight sanity checks. For the QFT subroutine, additional validation comes from cross checking its outputs against closed form mathematical solutions, and against implementations in other languages.

4.3. Numeric assertion checks for

composing gates with iterations

From the basic subroutines, programmers typically compose the subroutines into quantum programs using patterns including iterations, recursion, and mirroring. Here we focus on iterations, a pattern commonly invoked in code related to the QFT for the purpose of manipulating qubits that represent numbers. Our tool can catch bugs in iteration code using assertions on integer inputs to and outputs from subroutines.

Bug type 3: Incorrect composition of operations using iteration.

Now that we have validated code for the controlled rotation and QFT subroutines, the next more complex subroutine is the controlled adder, which is itself a subroutine for the modular exponentiation part of Shor’s algorithm (bottom module in Figure 2). Listing 2 shows the iteration code for the constant-value adder, showing tricky places in Lines 8 through 11 where bugs can crop up. These possible bugs include indexing errors in the two-dimensional loop, bit shifting errors, endian confusion, and mistakes in rotation angles.333One of the trickiest aspects of quantum programming is properly keeping track of how quantum variables map to qubit assignments. One way to prevent bugs altogether in this kind of code is to introduce QC data types for numbers, providing greater abstraction than working with raw qubits. For example, ProjectQ has quantum integer data types (Steiger et al., 2018), while Q# (Svore et al., 2018) and Quipper (Green et al., 2013; Valiron et al., 2015) offer both big endian and little endian versions of subroutines involving iterations. These QC data types permit useful operators (e.g., checking for equality) that help with debugging and writing assertions.

Defense type 3: Assertion checks for classical intermediate states.

Our tool’s assertions on classical integer values allows for unit testing of code that involve iterations. As an example in Listing 3, programmers can write assertions on the inputs (Line 15) and outputs (Line 24) of the controlled adder subroutine. With these assertions programmers can catch coding mistakes made in its constituent subroutines. For example the bug involving the incorrect version of the rotation operation in Table 1 is caught here when the output assertion returns $p\mathrm{-value}=0.0$, indicating the addition did not work as expected, due to a bug inside the controlled adder.

4.4. Entanglement assertion checks for

composing gates with recursion

The next two types of bugs in quantum programs have to do with two more ways to compose basic operations, both of which have to do with the interaction between quantum variables; i.e., between two or more sets of qubits. That is in contrast to the previous two types of bugs in basic operations and iterating operations, which generally act on single variables (where the variables may comprise multiple qubits).

In quantum computing, the interaction between variables takes place through entanglement. For example, in Figure 2, the upper and lower registers interact when they are entangled through the controlled modular exponentiation operation. If two variables are entangled when a quantum computer measures them, the classical values that they collapse to will be correlated. Using statistical tests on the measurement results, programmers can write assertions to check whether variables are entangled as expected.

Bug type 4: Incorrect composition of operations using recursion.

Entanglement is achieved using controlled operations, which is a common pattern in quantum programs that involves performing operations (e.g., modular multiply), contingent on a set of qubits known as control qubits. These controlled operations correspond to using recursion to compose basic operations. A multiply-controlled rotation, for example, is just a controlled rotation that is itself controlled by other qubits (Figure 4).

The process of coding recursive operation patterns may introduce bugs. That is because quantum algorithms often need varying numbers of control qubits in different parts of the algorithm, leading to replicated code from multiple versions of the same subroutine differing only by the number of control qubits. An example appears in Listing 2, where the addition operation is contingent on control qubits taken as parameters in Lines 4 and 5. Depending on how many control qubits are needed, the switch statement in Lines 12 through 15 applies the correct operation. The specific bug we are going to demonstrate catching next is if a programmer made a mistake in Line 15, where they accidentally use ctrl1 twice instead of ctrl0, causing a mistake in how the control qubits are routed.

Defense type 4: Assertion checks for entangled intermediate states.

Programmers can check for these types of bugs in recursive code patterns for controlled operations using entanglement assertions, a new kind of quantum assertion that we introduce to test for dependence between measured values.

As a very simple example, we show how the entanglement assertion check works on the simplest example of entangled states. In the Bell state creation circuit we showed in Figure 1, the state of the two qubits $Q$ in location (D) of that diagram are in a Bell state, a minimal example of an entangled state between qubits. The measurement results $m_{0}$ and $m_{1}$ are maximally correlated—either both return ‘0’ or both return ‘1’. Using such observations one can build a contingency table:

Probability $m_{0}$ measurement

0

1

$m_{1}$ 0

1/2

0

measurement 1

0

1/2

Next we again use a chi-square statistical test on the table to determine a contingency coefficient. If the $p$-value is small ($\leq 0.05$), as is the case for this table, then the test rejects the null hypothesis and concludes the observations must be correlated, and therefore the quantum variables were entangled when they were measured. On the other hand, if the $p$-value fails to be significant, then the observations are consistent with the variables being independent and unentangled.

These entanglement assertions are powerful tools for catching bugs such as our example bug of mistaken control qubits in the controlled adder subroutine. Our tool catches the bug using entanglement assertions in the controlled modular multiplier test harness, shown in Listing 4. To prepare the contingency table, the programmer only needs to identify pairs of quantum variables that should be entangled with each other using the assert_entangled statement (Line 37), which takes four parameters specifying the control and target quantum variables and their bitwidths. Then, our debugging tool keeps track of which qubits those specified variables correspond to. The simulator then does an early measurement of the qubits for both variables. The debugging tool then maps the measurement results into columns and rows of a contingency table automatically, and a chi-square test checks to make sure the control qubits have an effect on whether the multiplier acts on the target qubits.

If the controlled add operation is bug-free, with the control qubits correctly routed, the first assertion returns $p\mathrm{-value}=0.0005$ for an ensemble size of 16, indicating the control and target register values are entangled at the point of the assertion. That means that whichever way the control qubit collapses out of its superposition state, it correctly controls whether the multiplication works on the target register. On the other hand, if the control qubits are routed incorrectly, the first assertion returns $p\mathrm{-value}=0.121$ for an ensemble size of 16. This indicates the control register value is not correctly toggling the operation of the multiplier, hinting the bug must be somewhere inside the multiplier implementation.

4.5. Product state postcondition assertions for

composing gates with mirroring

Contingency table analysis is also useful for checking for the third and final kind of pattern in quantum programs, the correct mirroring of operations. The reason this pattern appears in quantum programs is to allocate and deallocate qubits within a quantum subroutine, analogous to the allocation and garbage collection of memory in classical programs. For example, the Shor’s algorithm in Figure 2 can be seen as allocating the bottom register of qubits (known as ancillary qubits) in the left half of the algorithm, performing the modular exponentiation, and then deallocating the bottom register of qubits in the right half of the algorithm. Product state assertions validate that the deallocation of these ancillary qubits is done correctly.

Bug type 5: Incorrect composition of operations using mirroring.

In order to garbage collect ancillary qubits in quantum programs, programmers need to reverse all the operations they applied to the qubits.

The reason programmers have to do so is because garbage collection is different in quantum computing compared to that in classical computing. In classical computing, programmers can simply mark any memory as unneeded in order to free it, and that memory would be rewritten some time later in program execution. But in quantum computing, qubits can be entangled and therefore cannot be treated as independent pieces of information. Suppose a program is done with using the lower register in Figure 2, but they remain entangled with the upper register qubits. Then anything that happens to the ancillary qubits, such as measurement, re-initialization, or lapsing into incoherence, can have unintended effects on the output qubits in the upper register that the program user does care about.

To prevent these unintended side effects, programmers have to carefully undo any entanglement they have built up between qubits. To do this, programmers perform inverse operations in backward order from the order they originally performed them. This process is called uncomputation (Kaye et al., 2007; Nielsen and Chuang, 2011; Häner et al., 2018). After uncomputation, ancillary qubits should be properly untangled from the rest of the program state, and are truly ready for reuse.

This process of uncomputation can be tricky if done manually. Take for example the controlled adder subroutine shown in Listing 2. Uncomputing the addition operation would need an inverse adder counterpart to the controlled adder. The code for the inverse adder would have each of the iterations in Lines 8 and 9 iterated in reverse order, and would have the rotation angles used in Lines 13 through 15 negated. Bugs in these inverse operations would impact the qubit deallocation process.

Defense type 5: Assertion checks for product state postconditions.

As a counterpart to entanglement assertions, our tool offers product state assertions to make sure that ancillary qubits and output qubits are in a product state, meaning they have no entanglement. This kind of assertion would make sure that code for the pattern of mirroring operations is correct.

We demonstrate the use of product state assertions also in Listing 4. Following the controlled modular multiplier in Line 35, the program reverses that operation in Line 41. The way the program invokes the inverse operation in Line 41 is by multiplying by the modular inverse. In the example here, $7\times 13\equiv 1\;\mathrm{mod}\;15$, so multiplying by $a^{-1}=13$ inverts the operation of multiplying by $a=7$. With the correct inverse computation, the assert_product statement in Line 43 returns $p\mathrm{-value}=1.0$, consistent with no entanglement between the upper control register and the bottom target register, indicating the bottom register is properly deallocated.

If on the other hand the program mistakenly multiplies by any number that is not the modular inverse, for example $a^{-1}=12$, then the assertion returns $p\mathrm{-value}=0.0005$ (for an ensemble size of 16) indicating the two registers are still incorrectly entangled, meaning the bottom register was not correctly deallocated, which hints to a bug in the mirrored code.

4.6. Classical postcondition assertions on

deallocated ancillary qubits

Finally, we are ready to run the Shor’s algorithm in an overall integration test. To run Shor’s algorithm, the programmer has to feed the algorithm pairs of modular inverse numbers as its input. For example, Table 2 shows the input pairs for factoring 15, using 7 as a trial divisor. Then, the algorithm should return 0, 2, 4, or 6, each with equal probability, from measuring the upper register (Nielsen and Chuang, 2011, p. 235). These numbers would go into a classical post-processing algorithm to find the factors of $3\times 5=15$.

Typically, programmers would only measure the upper register of qubits (Figure 2) that carry program output, and ignore the bottom register of qubits as they are merely ancillary qubits and should carry no information. However, when a programmer is debugging a quantum program, these ancillary qubits often carry useful side channel information that informs the programmer whether the main outputs are valid. Our tool checks for this information using classical assertions on the expected values for these deallocated ancillary qubits.

Bug type 6: Incorrect classical input parameters.

The final bug we study for Shor’s algorithm stems from giving wrong input parameters to an otherwise correctly written quantum program. These mistakes can be difficult to debug, even though the bug is entirely in the classical inputs to the algorithm.

The specific mistake is the programmer supplies wrong pairs of numbers as modular inverses for the algorithm. Instead of using $(a,a^{-1})=(7,13)$ for the first iteration in Table 2, the programmers gives a wrong pair of numbers $(7,12)$. We show our tool can debug this problem using assertions.

Defense type 6: Assertion checks for classical postconditions.

The outputs of Shor’s algorithm for this incorrect pair of inputs is recorded in Table 3. The table is a contingency table showing the joint probability for the output measurement and the ancillary qubit measurement. The table shows the ancillary qubits collapse to a non-zero value with probability $1/2$, which is incorrect because they should always return to their initial value of [math] after appropriate uncomputation. This symptom is to be expected because the incorrect pair of modular inverses fed to the algorithm has caused incorrect inversion of the multiplication operation inside the algorithm.

The programmer can use a classical assertion as a postcondition check on the deallocated ancillary qubits. The program should assert that the ancillary qubits should return their initial value of [math]. If the postcondition assertion fails, the programmer knows there was a bug in the deallocation of qubits and therefore the outputs may be wrong. If the postcondition succeeds, then the Shor’s factoring algorithm returns valid outputs.

5. QC program debugging across algorithm primitives

This section shifts focus away from the Shor’s algorithm case study and presents two additional debugging case studies. The goal is to understand whether the debugging techniques for Shor’s algorithm generally apply to other classes of algorithms.

In the Shor’s case study, we argued how the structure of the algorithm code guides the placement of assertions. Our methodology for debugging the algorithm was to bring up the subroutines from unit tests to full integration tests. We used assertions to check for preconditions, intermediate states, and postconditions of subroutines. Furthermore the code patterns of how subroutines are composed further guided what assertions to use. A natural question is whether that rigorous methodology is helpful for debugging other algorithms.

Many different quantum algorithms rely on a handful of QC algorithm primitives to get speedups relative to classical algorithms (Coles et al., 2018; Montanaro, 2016; Mosca, 2009). These algorithm primitives are akin to algorithm kernels in the context of classical algorithms. Each algorithm type has distinct pitfalls and features that lead to distinct bugs and possible defenses.

This section covers two more algorithms that use completely different algorithm primitives. The first is Grover’s database search algorithm based on the amplitude amplification primitive. The second is a quantum chemistry problem that uses quantum operations to simulate a physical system. This represents a broad selection of different quantum algorithm primitives.

While we have not covered in this paper some algorithm primitives (such as adiabatic algorithms, approximate optimization algorithms, and much less prominent primitives such as quantum random walks), the three areas we have covered represent the most important and well-studied algorithm classes.

5.1. Case study: Grover’s database search

This section uses the Grover’s benchmark to discuss how language syntax support for reversible computation and controlled operations guides placement of assertions.

5.1.1. Language support for placement of entanglement assertions

Higher-level quantum programming language features can help automatically place assert_entangled and assert_product assertions. We concentrate on the placement of these two assertion types because they are assertions on the relationship between two or more quantum variables. As such they are powerful debugging tools, but they also need the most programmer insight to correctly place them.

As we discussed in Sections 4.4 and 4.5, entanglement assertions are closely related to the quantum program patterns of recursion and mirroring. In the Scaffold language, these patterns are not explicitly captured by the C-style syntax, but in higher-level quantum programming languages, such as ProjectQ (Steiger et al., 2018) and Q# (Svore et al., 2018), these patterns are essential to the language design. With these language features, the placement of entanglement assertions becomes as natural as placing precondition and postcondition assertions.

5.1.2. The Grover’s algorithm for database search

The Grover’s search algorithm finds an entry that matches search criteria, among an input data set of size $N$, with a time cost on the order of $\sqrt{N}$. That represents a polynomial speedup relative to the linear time cost in a classical computer (Grover, 2001).

The Grover’s algorithm comprises three parts. First, the input qubits representing the indices of the matching entries are put in a state of superposition, akin to querying all entries at once. A superposition assertion (Section 4.1) helps certify that this algorithm precondition is satisfied. Second, the queries are put through a subroutine that checks for the search criteria. In our case study, our criteria is to find the square root of a number in a Galois field of two elements, a simple abstract algebra setting. Finally in the critical third step, the amplitude amplification algorithm primitive amplifies the index that matches the criteria while damping out those that do not.

5.1.3. Entanglement program patterns in the amplitude amplification subroutine

Table 4 shows the reversible computation and controlled operations program patterns coded in two quantum programming languages Scaffold (JavadiAbhari et al., 2014) and ProjectQ (Steiger et al., 2018). The ProjectQ language has syntax support for reversible computation that automatically mirrors and inverts sequences of operations. Likewise, syntax support for controlled operations automatically allocates the ancillary qubits needed for controlled operations.

These higher-level language support for these patterns allows automatic placement of assertions: the controlled operation statement in rows 3 through 5 indicates that register q should be entangled in row 4, so it would be the right place to place an entanglement assertion. Furthermore the compute-uncompute pattern in Rows 2 and 6 hints at a product state assertion at the end of uncomputation.

5.2. Case study: Quantum chemistry

Next, we discuss our experience building up and debugging a simple quantum chemistry program. Quantum chemistry problems entail finding properties of molecules from theoretical first principles (McArdle et al., 2018; Olson et al., 2017). Researchers anticipate these will be the first applications for QC due to the relatively few number of qubits they need to surpass classical computer algorithms. Debugging these problems is distinctively challenging, due to the importance of getting a large number of classical input parameters all correct, and because of the dearth of physically meaningful intermediate states we can check in the course of algorithm execution.

5.2.1. Classical input parameters

A key part of quantum chemistry programs is in correctly building up a Hamiltonian subroutine that simulates inter-electron forces. The procedure for doing this was laid out in detail by Whitfield (Whitfield et al., 2011). We followed this procedure to create a subroutine for simulating the hydrogen molecule, but we needed additional cross-validation from several other sources to get a bug-free subroutine (Wecker et al., 2014). These resources include raw chemistry data found in open source repositories for the LIQUi—¿ framework444https://github.com/StationQ/Liquid/blob/master/Samples/h2_sto3g_4.dat . The final parameters for actual operations on qubits were validated against a follow-up paper (Seeley et al., 2012) and an implementation in the QISKit framework555https://github.com/Qiskit/aqua/blob/master/test/H2-0.735.json. Because the procedure for preparing these quantum chemistry models involves many steps and needs domain expertise, arguably this step in preparing classical input parameters is the hardest aspect to debug.

Once the Hamiltonian subroutine is built, we can use the model in a variety of quantum algorithms spanning different primitives. These include phase estimation (an application of quantum Fourier transforms) (Patil et al., 2014), variational quantum eigensolvers (Peruzzo et al., 2014), and adiabatic algorithms (Barends et al., 2016). In this case study, we use iterative phase estimation to find the ground state energy of our $\mathrm{H_{2}}$ model, validating results published by Lanyon (Lanyon et al., 2010).

5.2.2. Assertions on quantum intial values and final states

The correct preparation of qubit initial values stands out as another challenging aspect of debugging quantum chemistry QC programs. Incorrect initial values would cause the program to find solutions to different problems altogether. In this quantum chemistry problem, the initial values control the locations of the two electrons in $\mathrm{H_{2}}$. As shown in Table 5, precondition assertions check the qubit assignment for finding the ground energy of $\mathrm{H_{2}}$, while other assignments lead to results for other energy levels.

The symmetry of $\mathrm{H_{2}}$ allows us to perform a sanity check, to make sure the Hamiltonian and the iterative phase estimation subroutines are working correctly. Though there are six ways to assign two electrons to four locations, there are in fact only four distinct energy levels, as shown in the experimental data (Table 5). Postcondition assertions are useful for checking that the two different ways to obtain E1 (and E2) give the same energy levels. These assertions validate that the model correctly preserves symmetry.

5.2.3. Assertions on intermediate algorithm progress

Unlike the other two case studies in this paper, the debugging process for the quantum chemistry benchmark is coarse-grained. That is because the Hamiltonian subroutine is a monolithic block of code whose components do not have obvious expected outputs—its components represent pair-wise electron interactions, and do not have inherent physical meaning. So how do we debug this program? The preconditions in the last subsection make sure the inputs to the algorithm are correct; the other observable state we have for debugging is to check the behavior of the algorithm as a whole.

In this quantum chemistry program, we can check for two types of overall algorithm behavior. One is the solution should converge to a steady value as finer Trotter time steps (a kind of numerical approximation) are chosen; a lack of this type of convergence indicates a bug in the Hamiltonian subroutine. The other algorithm behavior is when we vary the precision of the phase estimation algorithm, the most significant bits of the measurement output sequences should be the same—in other words, rounding the output of a high-precision experiment should yield the same output as a lower-precision experiment. a lack of this convergence indicates a bug in the iterative phase estimation subroutine. These checks for expected algorithm progress also apply to other algorithms.

6. Related work

Approaches to writing correct quantum programs range from formal methods to less-formal pragmatic methods, much like in classical programming. Most of the prior work in quantum program correctness has been in formal methods–e.g., using theorem provers and type checking to verify programs correctly match algorithm specifications (Ying, 2012; Ying et al., 2017; Hung et al., 2019; Rand, 2018; Paykin et al., 2017). Such verification techniques consider the correctness problem from a top-down perspective. While useful, formal methods should not be the only approach to correct programs; more traditional debugging strategies are also useful. This work considers the possibility of using pragmatic assertion checks to build code bottom-up from exhaustive unit tests up through integration testing.

Quantum program assertions exist in several quantum programming languages, though not in the same capacity as the statistical assertions presented in this work. First, the Quipper language (Green et al., 2013) features assertive termination, which allows the programmer to annotate known program state, in order to drive compiler optimizations. Their use relies on the programmer to write correct code and assertions, and cannot be used as postcondition checks. Second, the Q# programming language (Svore et al., 2018) allows programmers to write assertions on classical, integer states. These assertions are then checked during the simulation of the quantum programs. This work extends that set of assertions with assertions on superposition and entanglement states.

The set of quantum states considered in these assertions, namely classical, superposition, and entangled states, are a subset of possible quantum states. In the general case we need quantum phase estimation, quantum state tomography, and quantum process tomography to be able to examine general quantum states (Nielsen and Chuang, 2011). However, those processes are extremely costly and cannot be used as efficient assertion checks.

7. Conclusion

For the first time, we have access to program benchmarks for several major areas of quantum algorithms, along with input datasets and outputs that are detailed enough to permit detailed debugging and cross-validation. Using our experience in building up and debugging these programs, we presented in this paper a strategy for deploying and checking quantum program assertions based on statistical tests. Drawing on the structure of quantum programs, we point to where and how program bugs may arise, and point to how the presented assertions can catch them.

Acknowledgements.

This work is funded in part by EPiQC, an NSF Expedition in Computing, under grant 1730082.