Benchmarking quantum processors with a single qubit

TL;DR

This paper benchmarks IBM quantum processors using the DQC1 algorithm to evaluate performance and error sources, revealing circuit length and width as key factors affecting accuracy.

Contribution

It introduces a benchmarking method for quantum processors using DQC1, analyzing how circuit parameters influence performance and error behavior.

Findings

Circuit length is the main limiting factor for performance.

Wider circuits perform better at the same gate count due to error randomization.

Both random and systematic errors increase with circuit complexity.

Abstract

The first generation of small noisy quantum processors have recently become available to non-specialists who are not required to understand specifics of the physical platforms and, in particular, the types and sources of noise. As such, it is useful to benchmark the performance of such computers against specific tasks that may be of interest to users, ideally keeping both the circuit depth and width as free parameters. Here we benchmark the IBM Quantum Experience using the Deterministic Quantum Computing with 1 qubit (DQC1) algorithm originally proposed by Knill and Laflamme in the context of liquid state NMR. In the first set of experiments we use DQC1 as a trace estimation algorithm to produce visibility plots. In the second set we use this trace estimation algorithm to distinguish between knots, a classically difficult task which is known to be complete for DQC1. Our results indicate that the main limiting factor is the length of the circuit, and that both random and systematic errors become an issue when the gate count increases. Surprisingly, we find that at the same gate count wider circuits perform better, probably due to randomization of coherent errors.

Tables

Table 1. Table 1: Assuming only depolarizing noise, we fit a decay model f ​ ( x ) = a ​ e − x / τ 𝑓 𝑥 𝑎 superscript 𝑒 𝑥 𝜏 f(x)=ae^{-x/\tau} to the data in Fig.  4 (linear fit to the logarithm of the data).

Qubits	$\tau$	$R^2=1-\frac{{\sum }_i{\left(y_i-f_i\right)}^2}{{\sum }_i{\left(y_i-\overline{y}\right)}^2}$
Q5	24.50	0.933
Q7	25.81	0.995
Q11	28.91	0.695
Q5, Q7, Q11	30.75	0.997

Table 2. Table 2: Comparison of Jones polynomial estimates for implementations of different knots using the similar circuit depths. A large distance, relative to the error, indicates that two knots can be distinguished using the IBM QPU.

CNOTs in ${\sigma }_{23}^k$	Unitaries	J.P. dist. (Exp.)	J.P. dist. (Theory)
5	$\left|{\sigma }_{23}-{\sigma }_{12}^2\right|$	$1.42\pm 0.05$	2.15
5	$\left|{\sigma }_{23}-{\sigma }_{12}^3\right|$	$2.50\pm 0.08$	3.62
10	$\left|{\sigma }_{23}^2-{\sigma }_{12}^5\right|$	$0.66\pm 0.05$	1
15	$\left|{\sigma }_{23}^3-{\sigma }_{12}^7\right|$	$2.6\pm 0.2$	4.25
15	$\left|{\sigma }_{23}^3-{\sigma }_{12}^8\right|$	$2.0\pm 0.2$	3.24

Equations

$${\rho }_f^{(1)}=\frac{1}{2}\left(I_1+\frac{\text{Tr}U}{2^N}|1⟩⟨0|+\frac{\text{Tr}{U}^{†}}{2^N}|0⟩⟨1|\right)$$

$$U_{N=1}^{(l)}(\theta )=U_1(\theta )(U_1(\theta {)}^{†}{U}_1(\theta ){)}^{l-1}$$

$$U_{N=3}^{(l)}(\theta )=U_3(\theta )(U_3(\theta {)}^{†}{U}_3(\theta ){)}^{l-1}$$

$$\text{Vis}=\underset{\theta }{\max }⟨{\sigma }_x⟩.$$

$${\sigma }_{12}=\left(\begin{array}{cccc}a& 0& 0& 0\\ 0& b& 0& 0\\ 0& 0& a& 0\\ 0& 0& 0& 1\end{array}\right),{\sigma }_{23}=\left(\begin{array}{cccc}e& d& 0& 0\\ d& c& 0& 0\\ 0& 0& a& 0\\ 0& 0& 0& 1\end{array}\right),$$

$$\text{WTr}U$$

$$V_U(t=e^{2\pi i/5})=(-(e^{2\pi i/5}{)}^4{)}^{3w}\times \frac{1}{\varphi }\text{WTr}U,$$

Full text

1 Introduction

Small noisy quantum processors can now be implemented in various platforms and architectures including superconducting circuits [4, 26, 21], trapped ions [9], optics [24] and NMR [16]. These and other near-future processors are not expected to be universal for quantum computation [22] and need to be benchmarked in tasks that are suitable for noisy processors with little or no error correction. The deterministic quantum computing with one qubit (DQC1) algorithm, which was originally developed for noisy NMR quantum processors, offers a good way to benchmark these processors. In this work we benchmark two IBM quantum processors, first using simple DQC1 circuits to calculate the trace of a unitary, and then, in a specific task, using DQC1 to distinguish between knots.

The experiments used between 3 and 8 qubits and were initially run on the IBM Q 16 Rüschlikon [3] and later on the IBM Q 14 Melbourne[2]. The first set of experiments (Sec. 4) involved the estimation of the normalized trace of 1 and 3 qubit unitaries. The results allow us to make some general statements about the noise in the circuit, in particular depolarizing noise and systematic (coherent) errors. Somewhat surprisingly, the performance of the 3 qubit algorithms as a function of the number of gates was better than the 1 qubit algorithms, most likely due to the reduction in correlated noise when the gates act on different qubits. In the second set of experiments, we used the DQC1 algorithm to evaluate various Jones polynomials (Sec. 5). The results show that while the evaluated Jones polynomials tend to be far from theoretical values, the errors are consistent for the different circuits. This implies that the processors can be used to distinguish between various knots made by closing a braid of up to 3 strands, as long as the evaluations are run at approximately the same time (i.e., not days apart) using the same subset of qubits, such that systematic errors in gate operations remain approximately the same from run to run.

2 DQC1

The DQC1 model was originally proposed by Knill and Laflamme [12] in the context of room temperature, liquid state NMR quantum computing where the initial (thermal) state $\rho_{i}$ is very noisy. As a consequence of the noise, the signal-to-noise ratios in the readout are small and the computation is done on an ensemble with ensemble readout, i.e., the result is an estimate of the expectation value of some observable. Knill and Laflamme noted that in an $N+1$ qubit NMR processor it is possible to prepare an initial state of the type $\rho_{i}=[\alpha\ket{0}\bra{0}+(1-\alpha)I_{1}]\otimes I_{N}$ (where $I_{n}=\frac{1}{2^{n}}\mathbbm{1}_{n}$ is the $n$ qubit maximally mixed state) efficiently111The initialization procedure works for some fixed $\alpha\ll 1$ that depends on the parameters of the experiment, and does not scale badly with $N$.. Under the assumption that the evolution is given by a unitary operator $V$, the final readout on the first qubit will be $\operatorname{Tr}(V\rho_{i}V^{\dagger}\sigma_{k}^{(1)})=\alpha\operatorname{Tr}(V\left(\ket{0}\bra{0}\otimes I_{N}\right)V^{\dagger}\sigma_{k}^{(1)})$ where $\sigma_{k}^{(1)}$ is a Pauli operator on the first qubit and $k\in\{x,y,z\}$. Noting that the polarization parameter $\alpha$ is simply used to re-scale the expectation value, it is possible to assume $\alpha=1$ without loss of generality, as we will do throughout this work. Under this assumption the first qubit is initially pure, or “clean”, while the other qubits are completely mixed. This model is therefore sometimes called the ‘one clean qubit’ model [25].

In the DQC1 model the classical input describes the unitary operator $V$ which is assumed to have an efficient description, i.e., it can be decomposed into a (polynomial in $N$) sequence of one and two qubit gates. It is common to further restrict $V$ to a Hadamard operator on the first qubit, followed by a controlled unitary from the first qubit, $U$, targeting all other qubits, i.e., $V=\left(\ket{0}\bra{0}\mathbbm{1}_{N}+\ket{1}\bra{1}U\right)H^{(1)}$. Here $U$ is an $N$ qubit unitary with an efficient classical description and $H^{(1)}$ is a Hadamard operator on the first qubit (this is sometimes called cDQC1 [7]). In this restricted model, the state of the first qubit at the end of the computation (before readout) is given by

[TABLE]

so that $\frac{1}{2^{N}}\operatorname{Tr}U=\langle\rho_{f}^{(1)}(\sigma_{x}+i\sigma_{y})\rangle$. That is, the model can be used to estimate the normalized trace of the unitary $U$. A number of results suggest that the trace estimation algorithm cannot be simulated efficiently by a classical computer, with some recent results including the use of DQC1 for parity learning [18], and a sampling version [17] which follows the standard definition above, but allows single shot readout.

Shor and Jordan [25] used the DQC1 model to define a computational complexity class. They then showed that the trace estimation algorithm is computationally equivalent to the full DQC1 model and furthermore showed that adding a small (at most logarithmic in $N$) number of pure qubits does not change the computational power of the model. They also showed that the estimation of the Jones polynomial for the trace closure of a braid at the fifth root of unity (a problem in knot theory, see Sec. 5 below) is DQC1 complete.

2.1 Noise in DQC1

The DQC1 model is designed to handle noisy initial states but, to the best of our knowledge, its performance under noisy dynamics has not been analyzed. An $N$ qubit unitary $V$ can generally be decomposed into a sequence of fundamental unitaries $\{W_{k}\}$ such that $V=\prod_{k}W_{k}$. Ideally, these fundamental unitaries correspond to gates that are physically implementable on the processor. But in practice, the gates are imperfect and errors that are often difficult to characterize degrade the computation[8, 27].

One fairly simple model is to assume depolarizing noise, where each gate is a probabilistic mixture of the desired unitary $W_{k}$ and a completely depolarizing channel. The ideal transformation $\rho\rightarrow W_{k}\rho W_{k}^{\dagger}$ of the state is replaced by $\rho\rightarrow\alpha_{k}W_{k}\rho W_{k}^{\dagger}+(1-\alpha_{k})\mathbbm{1}$ where $\alpha_{k}$ is the purity of the channel. All subsequent unitary operations and depolarizing channels leave the identity unchanged, so the full sequence will be $\alpha V\rho_{i}V^{\dagger}+(1-\alpha)\mathbbm{1}$ with $\alpha=\prod_{k}\alpha_{k}$.

The fact that the purity falls exponentially with the number of gates does not bode well for the computation. Standard quantum error correction methods rely on a supply of pure qubits so they are not suitable for DQC1. However, for our purposes, focusing on small or intermediate size processors, the issue of exponential noise might not be debilitating especially when one considers a logical implementation of the model using some clean physical qubits and single shot measurements to combat errors.

The relatively simple behavior of the DQC1 algorithm in the presence of depolarizing noise makes it a good tool for benchmarking against a depolarizing noise model. In our results below, we used the $R^{2}$ of a fit to the depolarizing noise model as a benchmark. The behavior of the circuits as a function of the number of gates provided evidence that the most significant source of error was a systematic error in the CNOT gates.

3 Implementation of the algorithm

We implemented DQC1 on IBM superconducting qubit quantum processing units (QPU) via a web-based application programming interface (API). The code is available online [1] and a technical descriptions of the processors can be found in Ref. [4]. All results described in the present work are limited to data obtained via the web API, and not direct physical access to IBM hardware. Basic tests to benchmark DQC1 performance on gate-based machines, described in Section 4, were executed on the 16-qubit “Rüschlikon” QPU. Application of DQC1 to a useful task (evaluation of Jones polynomials), described in Section 5, was executed on the 14-qubit “Melbourne” QPU.

A single qubit (shown in red in Fig. 2) was designated the “clean” qubit whereas another disjoint subset of qubits (shown in blue in Fig. 2) was chosen to be “noisy”. A gate-based QPU, however, is usually designed to operate with pure states under unitary evolution as much as possible. To prepare these “noisy” qubits in the $\left(\mathbb{I}/2\right)^{\otimes N}$ state, we used two techniques. In the first set of experiments we first entangled each qubit with an adjacent qubit to produce the (pure) Bell state $\left|\Phi^{+}\right>=(\left|00\right>+\left|11\right>)/\sqrt{2}$, and then ignored (or traced away) that adjacent qubit. This approach introduces a 2X overhead in the number of qubits required for state preparation of these “noisy” qubits. For the estimation of the Jones polynomial, we used bit flip on the “noisy” qubit for half the experiments and averaged over the results. This method can be modified for a multiple qubit scenario by randomly flipping all “noisy” qubits and averaging over the results.

4 Trace estimation on the quantum processor

We implemented the $N=1$ version of the trace estimation algorithm (Fig. 1) with

[TABLE]

where $l\geq 1$ is the number of repetitions and $U_{1}(\theta)=e^{-i\theta/2}\left|0\right>\left<0\right|+e^{i\theta/2}\left|1\right>\left<1\right|$ (see top RHS of Fig. 3). We also implemented the $N=3$ version, replacing $U_{1}^{(l)}(\theta)$ with $U_{3}^{(l)}(\theta)=U_{1}^{(l)}(\theta)^{\otimes 3}$ so

[TABLE]

(see bottom RHS of Fig. 3).

The circuit was chosen to maximize contrast with respect to $\theta$ (i.e., $\operatorname{Tr}[U_{1}^{(l)}(0)]=1$ and $\operatorname{Tr}[U_{1}^{(l)}(\pi)]=-1$) for any value of $l$. Increasing $l$ merely introduces repetition of a gate sequence that should, logically, be equivalent to the identity. In practice, however, gate errors and noise means increasing $l$ yields noisier outputs. Results for a final measurement of $\sigma_{x}$, $\sigma_{y}$ and $\sigma_{z}$ are shown in Fig. 3.

As expected, the results deviate further from the ideal as we go to higher gate counts. The reduction of the absolute values in the $\sigma_{x}$ plots can be attributed to depolarizing noise, however the fact that the shape changes in all three plots (and in particular the deviations from 0 in $\sigma_{y}$), indicate a coherent error, probably as a result of a systematic error in the CNOT gates. For a quantitative indication of the coherent errors (more precisely the deviation from depolarizing noise) we defined the visibility

[TABLE]

This function decays exponentially with the number of gates when imperfections are due to depolarizing noise. A plot of visibility as a function of the number of gates (Fig. 4) shows that this is clearly not the case for qubit 11 (paired with 5) where there is a spike in visibility around 20 CNOT gates. The other couplings show the expected qualitative behaviour, but a fit to a depolarizing noise model shows some deviations (see Table 1).

Generally, it is possible to convert biased noise channels into a depolarizing channels by adding some randomness and averaging. This is apparent when comparing the 1+1 and 1+3 qubit results. In Table 1 we see that fit for the 1+3 qubit result is better than each of the individual results. This is most likely a result of averaging over different coherent errors for the 3 different pairs of interacting qubits. We note that in general this is not an indication of better performance overall. Though the exponential depolarizing rate appears slower in the 1+3 qubit case, the circuit will have to be 3 times longer to perform a similar task.

Running the experiment at different times produced different results, in particular the systematic (coherent) errors were not consistent over long periods of time. Results for the expectation value of $\sigma_{y}$ taken on the same pair of qubits 5 days apart are plotted in Fig. 5. Since the theoretical expectation value should be constant ($\langle\sigma_{y}\rangle=0$) the plots are a good indication of coherent errors which, even at a circuit depth of 6 CNOT gates, produce a visibly different plot.

5 Distinguishing knots with Jones polynomials

The task of identifying whether two knots (smooth closed curves in $\mathbb{R}^{3}$) are topologically equivalent has implications beyond mathematics, reaching into statistical mechanics, quantum field theory and quantum gravity [6, 23]. Knots can be faithfully represented in two-dimensional pictures, and so the task can be recast as determining whether two pictures of knots can be made equal using transformations called Reidemeister moves. This task is computationally intensive and even in the simplest case, identifying a knot as the unknot, there is no known efficient solution [14]. Here we consider a particular type of knot, the trace closure of a 3-strand braid, which can be represented by drawing a two-dimensional braid and closing each end at the bottom with the associated strand at the top (right-most to right-most etc., see Fig. 6(c). Another type of knot, the plat closure, is constructed by connecting adjacent strand ends at the top and at the bottom.

The Jones polynomial is a complex function invariant for oriented knots [11, 10], which allows one to distinguish one knot from another. However, constructing the polynomial for a given two-dimensional picture of a knot is not trivial. The number of terms in the polynomial (before simplification) scales exponentially with the number of strand crossings. Evaluating the Jones polynomial at a single point would give sufficient information to tell if two knots are different. Equivalent knots must have the same Jones polynomial value at any given point, whereas knots that are not the same might not. Approximating the value of the Jones polynomial at $e^{2\pi i/5}$ is a particularly interesting task for quantum computers. If one takes the plat closure of a braid, rather than the trace closure, the task is known to be BQP-complete [5].

In 2007 Shor and Jordan [25] showed that approximating the polynomial at $e^{2\pi i/5}$ for the trace closure of a braid is a complete task for DQC1. Passante et al. [19] demonstrated this task in a four-qubit liquid-state NMR processor, studying knots with four strands and multiple crossings. Here we use DQC1 to approximate the Jones polynomial for knots of three strands for the purpose of benchmarking the IBM Q 14 Melbourne quantum processor. We consider knots which are constructed by taking the trace closure of braid words up to 9 crossings (see Fig. 6). We study the same knots constructed by multiple iterations of either the $\sigma_{12}$ crossing (first strand over the second), or the $\sigma_{23}$ crossing (second over the third). Since approximating the values of Jones polynomials with a noisy machine is difficult, we are content with the ability to classify knots as different when they are indeed different. As in the previous sections, we do not perform any type of error mitigation during the computation or in post-processing, apart from simplifying the circuit to require fewer gates.

Following the treatment in Ref. [19], the unitary used in the DQC1 protocol is related to the braid through the Fibonacci representation. For a three-strand braid, the $\sigma_{12}$ and $\sigma_{23}$ unitaries are given by

[TABLE]

where $a=e^{3\pi i/5}$, $b=e^{-4\pi i/5}$, $c=\frac{b}{\phi^{2}}+\frac{a}{\phi}$, $d=\frac{b-a}{\phi^{3/2}}$, $e=\frac{b}{\phi}+\frac{a}{\phi^{2}}$, and $\phi=(1+\sqrt{5})/2$. For braids with $n$ strands we require $m\times m$ sized unitaries where $m$ is the $n^{\text{th}}$ number in the Fibonacci sequence. For 3 strands the unitaries map between 3 states, requiring 2 qubits. This results in an unused portion of the Hilbert space – the $\ket{11}$ state is not used for the approximation and adds a constant term to the trace which is straightforwardly dealt with.

To implement DQC1 we must construct controlled versions of the braid word unitaries. However, a single controlled-$\sigma_{23}$ operation requires approximately 50 CNOT gates, meaning any braid word would be prohibitively long on current quantum processors. Indeed, Sec. 4 shows that $\left\langle\sigma_{x}\right\rangle$ and $\left\langle\sigma_{y}\right\rangle$ on the clean qubit would decay substantially after just one controlled-$\sigma_{23}$ gate. To simplify the problem we use the fact that both $\sigma_{12}$ and $\sigma_{23}$ are block diagonal, and so any braid word will also be. We perform a controlled version of each block of a braid word in separate experiments, measure their traces via the clean qubit, and combine them afterwards. The controlled implementations of the blocks require substantially fewer CNOT gates: $\sigma_{12}^{(u)}$, $\sigma_{23}^{(u)}$, and $\sigma_{12}^{(l)}=\sigma_{23}^{(l)}$, where $u$ $(l)$ refers to the upper (lower) block of the unitary, can be performed with 2, 5 and 2 CNOT gates respectively. The braidword for the trefoil knot outlined in Fig. 6, $\sigma_{12}^{3}$, requires 6 CNOT gates for each of the upper and lower blocks. Performing the braidword $\sigma_{23}^{3}$, which gives the same knot upon taking the trace closure, requires 15 CNOT gates for the upper block and 6 for the lower.

Having used the DQC1 protocol to measure the trace of each block of a braid word unitary, we then estimate the value of Jones polynomial at the fifth root of unity for each knot. To do this we combine the measurements on each block of the unitary, e.g., $U=\sigma_{12}^{3}$, to find the weighted trace of the braid word. First, we subtract off the contribution to the trace of the lower block, $U^{(l)}$, from the $\ket{11}$ state. We then add the traces of the two blocks together while weighting the upper block by a factor of $\phi$ [19, 20],

[TABLE]

We then calculate the Jones polynomial value as

[TABLE]

where $w$ is the writhe of the knot, defined as the difference between left-over-right crossings ($\sigma_{12}$ and $\sigma_{23}$) and right-over-left crossings ($\sigma^{\dagger}_{12}$ and $\sigma^{\dagger}_{23}$). In this work we only consider knots with $w>0$.

In Fig. 8 we plot one set of results of the Jones polynomial, $V_{U}(e^{2\pi i/5})$, estimation for knots with 0 to 9 crossings, constructed solely by either $\sigma_{12}$ or $\sigma_{23}$ crossings. We find that as the numbers of crossings are increased the estimated polynomials quickly deviate from theoretical values (see Fig. 9). This is to be expected from the studies in previous sections. As the circuit depth increases, the measured $\left\langle\sigma_{x}\right\rangle$ and $\left\langle\sigma_{y}\right\rangle$ on the clean qubit decay exponentially. This in turn causes the Jones polynomial values to tend to the origin on the complex plane with increasing circuit depths.

Though the deviation from the theoretical value is not ideal performance, the principle behind estimating the Jones polynomial is to distinguish between knots. In this spirit, we note that the distance between the two implementations of each knot (using either $\sigma_{12}$ or $\sigma_{23}$) remains relatively low compared to the distance from their theoretical value. Differences between the two experimental implementations of the same knot are likely driven by the significantly higher circuit depth for each $\sigma_{23}$ unitary (5 CNOTs vs. 2 CNOTs). Importantly, if we compare different knots that have the same circuit depth – e.g., $\sigma_{12}^{5}$ and $\sigma_{23}^{2}$ each use 10 CNOT gate for their upper blocks, and represent different knots – we see that they are largely distinguishable from one another when the gate count is low.

At higher gate counts the values are not only closer to the origin (as expected), but also behave qualitatively different from the theoretical results, for example in Fig. 9 the real part of the braid with 7 crossings should be more negative than that of 6 crossings, but in both implementations ($\sigma_{12}$ and $\sigma_{23}$) the knot with 7 crossings is positive while the knot with 6 is negative. Moreover these types of errors, while fairly consistent on a single run of the experiment, appear to be very different when the experiment is repeated later and/or on different qubits. In Fig. 10(a) we show the results for 0 and 3 crossings ($\sigma_{12}$) taken on all 18 different pairs of qubits. The results indicate that both the mean and the spread depend on the choice of qubits. In Fig. 10(b) we compare the results of the braids with 0 and 3 crossings ($\sigma_{12}$) at different times, where we chose the qubits pairs Q5,Q6 and Q4,Q5 for best performance. Even at relatively low gate counts, we observe deviations from one run of the experiment to the next. One consequence of these results is that there is no simple way to correct for errors in post-processing. Such corrections would have been possible if the dominant source of error was depolarization, in which case we could multiply the results by a factor that depends on the number of gates.

6 Discussion

As quantum computers become more available to the non-specialists, there is need for tools that can be used and understood by users who are not interested in the inner workings of the machine. Benchmarking protocols have usually been designed with the experimental physicist in mind, often to quantify performance in terms of noise per gate, and usually with an outlook towards fault tolerance [15, 13]. The benchmarking experiments we used in this work were designed with the end user of a small noisy quantum processor in mind. Using two protocols based on the DQC1 trace estimation algorithm we benchmarked IBM 14- and 16-qubit processors and showed how performance degrades with circuit depth, in one case for the useful task of distinguishing knots.

In the first set of experiments we looked at how visibility drops as a function of circuit depth, and showed that coherent errors can be particularly harmful. For example we noted a spike in visibility (Fig. 4) for the DQC1 experiment with Qubit 11 on the "Rüschlikon". We also observed an undesired buildup of an imaginary component (coherence in the $\sigma_{y}$ axis) as the gate count increased. This was evident as early as 10 CNOT gates in a 2-qubit experiment (Fig. 3). Surprisingly, we noted that the performance is not reduced (and was even enhanced on average) in a 4-qubit experiment compared to the 2-qubit experiments when results with a similar CNOT gate count were compared (Fig. 3 and Fig. 4). Finally, we saw that the results were not consistent when the same experiment was performed at different times (Fig. 5). However, the qualitative results regarding performance at different gate counts remained the same.

In the second set of experiments we used 2 qubits on the IBM "Melbourne" to estimate the Jones polynomial at the 5th root of unity for various knots. While the results deviated from theory (Fig. 9) it was possible to compare knots at low gate counts (Fig. 8). However, at higher gate counts both depolarizing and systematic errors start to dominate the results. While depolarizing noise can be countered by repeating the experiment more times and normalizing 222Repeating experiments is generally not a scalable technique., the systematic errors (which are not constant in time) are difficult to deal with even in a small circuit. We note that these errors prevented us from outperforming the 4-qubit liquid state NMR experiment [19].333However this is not an apples-to-apples comparison since (without fault tolerance) a 4 qubit machine is expected to outperform a 14 qubit machine in a $\leq 4$ qubit experiment.

The relatively simple benchmarking procedure leads us to the conclusion that at least from the end user’s perspective, a major issue with current small noisy quantum computers is constantly changing environmental conditions that leads to frequently changing systematic errors. While it is clear that this is a major engineering challenge for superconducting architectures due to the sensitivity to environmental conditions, it is worthwhile considering alternative architectures that may be more stable. A different approach might be a method to reduce systematic errors on the software side for example by using wider and shallower circuits or to turn these into statistical errors by using various randomization techniques.

Acknowledgements

We acknowledge use of the IBM Q for this work. The views expressed are those of the authors and do not reflect the official policy or position of IBM or the IBM Q team. This work was partially supported by the CIFAR “Quantum Information Science” programme and an NSERC grant “Experimental Quantum Information, Quantum Measurement, and Quantum Foundations With Entangled Photons and Ultracold Atoms” via Aephraim Steinberg’s research group. KBF acknowledges the NSERC PDF programme for funding.