On the correctness of Egalitarian Paxos

TL;DR

This paper examines a critical correctness issue in Egalitarian Paxos related to replica ballot switching, which can cause divergence and violate linearizability, highlighting the need for correction in the protocol's specification and implementation.

Contribution

It identifies a specific flaw in Egalitarian Paxos's handling of ballot dependencies that can lead to divergence and correctness violations.

Findings

The flaw can cause replicas to diverge.

The issue affects the linearizability of the protocol.

The paper suggests the need for protocol correction.

Abstract

This paper identifies a problem in both the TLA+ specification and the implementation of the Egalitarian Paxos protocol. It is related to how replicas switch from one ballot to another when computing the dependencies of a command. The problem may lead replicas to diverge and break the linearizability of the replicated service.

Full text

1 Introduction

State-machine replication (SMR) is a fundamental technique to build dependable services. With SMR, a service is replicated across a set of distributed processes. Replicas apply to their local copies the commands that access the replicated service following a total order. This order is implemented using a repeated agreement protocol, or consensus, on the next command to execute.

Some recent works [1, 2] observe that it is not necessary to build a total order over the commands submitted to the service. To maintain service consistency and provide the illusion of having no data replication, ordering non-commuting commands suffices. Egalitarian Paxos (EPaxos) [3] is a novel protocol built upon this insight. To construct the partial order, EPaxos replicas agree on the dependencies of each command submitted to the service. A replica executes commands according to the graph formed by these dependencies, that is, if $c$ depends on $d$, then $c$ is executed after $d$.

To agree on the dependencies of a command, EPaxos follows a common pattern: replicas successively join asynchronous ballots, and during a ballot, they try to make a decision.

Surprisingly, the ${\text{TLA}^{+}}$ specification and the Golang implementation of EPaxos use a single variable at each replica to track progress across ballots. This paper shows that this is not sufficient. We exhibit an admissible execution in which replicas disagree on the dependencies of a command, breaking consistency.

Outline

Section 2 recalls the traditional schema of repeated asynchronous ballots to reach consensus. Section 3 gives an overview of the EPaxos algorithm. Section 4 depicts our counter-example. Section 5 closes this paper. An appendix follows that contains the ${\text{TLA}^{+}}$ specification of the counter-example. The specification is also available online [4].

2 Solving consensus

The classical approach to solve consensus is to execute a sequence of asynchronous rounds, or ballots. Each ballot is identified with a natural, its ballot number. As usual, we refer to a ballot with its ballot number and assume that processes start the agreement at ballot [math].

During a ballot, a quorum of processes attempt to agree on some proposed value. To this end, each ballot is split into three distinct phases. Before it participates in a ballot, a process first joins it (the prepare phase). Then, it may vote for some proposed value (accept phase). A value is chosen at a ballot when a quorum of processes voted for it. A process decides some value once it knows that this value was chosen at some ballot (learn phase).

The size of a quorum depends on the time taken by the protocol to reach a decision in a ballot. Consider a system of $n=2f+1$ processes of which at most $f$ may fail-stop. In Paxos [5], any majority of the processes (that is, at least $f+1$ of them) is a quorum at every ballot. Differently, Fast Paxos [6] distinguishes fast and classic ballots. Whereas a majority of processes is a quorum for a classic ballot, a quorum of a fast ballot contains at least $f+\lfloor\frac{f+1}{2}\rfloor+1$ processes.

Reaching consensus requires to choose a unique value among the proposals. To this end, the following invariant is maintained across ballots during an execution:

• (INV) if a value $v$ is chosen at some ballot $b$, then for every ballot $b^{\prime}\geq b$, if $u$ is chosen at $b^{\prime}$, $u=v$ holds.

To build this invariant, consensus algorithms commonly rely on the following assumptions.

• (A1)

A process can join a ballot $b$ only if it did not join some ballot $b^{\prime}>b$ previously.

• (A2)

A process may only cast a vote for the last ballot it has joined.

• (A3)

The vote of a process at a ballot is irrevocable. This means that if a process votes for some value $u$ at ballot $b$, it cannot later vote at ballot $b$ for some other value $v\neq u$.

• (A4)

The votes of any two processes at a ballot are identical, that is, if $p$ and $q$ vote at ballot $b$ for respectively $u$ and $v$, then $u=v$.

• (A5)

Consider two quorums $Q$ and $Q^{\prime}$ defined respectively at ballot $b$ and $b^{\prime}$. Then, the intersection of $Q$ and $Q^{\prime}$ is non-empty.

To maintain invariant (INV), the consensus protocol tracks the values chosen at prior ballots when it progresses to a new ballot. In detail, a process $p$ that aims at advancing to ballot $b$, first seals prior ballots. To this end, for every ballot $b^{\prime}<b$, and every quorum $Q$ at $b^{\prime}$, $p$ asks a process q{\makebox[7.7778pt][c]{\in}}Q to join $b$. By invariants (A1) and (A2), sealing prevents new value to be chosen at a lower ballot than $b$.

Once a quorum $Q$ of processes have joined ballot $b$, $p$ computes the values chosen prior to $b$. Assume that some value $v$ is chosen at ballot $b-1$ by a quorum $Q^{\prime}$. By invariants (A3-A5), $v$ is unique, and some process q{\makebox[7.7778pt][c]{\in}}Q\cap Q^{\prime} voted for $v$ at ballot $b-1$. Hence, process $p$ discovers $v$ when it inquiries the processes in $Q$ at the time they join ballot $b$.

Now if $v$ does not exist, by induction, $p$ must propose what was voted at ballot $b-2$, etc. In other words, $p$ proposes at ballot $b$ the value for which some process in $Q$ voted at the largest ballot prior to $b$. If no such value exists, $p$ is free to pick any value as its proposal. If (INV) is true up to ballot $b-1$, then the invariant is maintained at ballot $b$.

The proposal of $p$ is a so-called safe value. It maintains invariant (INV) at ballot $b$ by over-approximating what was chosen at prior ballots. In the literature [2], the construction of a safe value makes use of three variables at each process. The first one, denoted hereafter $\mathit{bal}$, is the last ballot joined so far by a process. The second variable $\mathit{vbal}$ is the last ballot at which the process voted for some value. Variable $\mathit{vval}$ contains the value that was voted at ballot $\mathit{vbal}$.

The ${\text{TLA}^{+}}$ specification of EPaxos and its implementation rely on not two ballot variables, as mentioned above, but a single one to reach an agreement on the dependencies of a command. In the remainder of this paper, we explain that this mechanism is flawed. To this end, we give an overview of the EPaxos protocol, then we exhibit an admissible run that breaks safety.

3 Overview of EPaxos

Egalitarian Paxos (EPaxos) is a recent protocol to implement state-machine replication and construct fault-tolerant distributed services. EPaxos is leaderless and it orders commands in a decentralized way without relying on a distinguished process. As in [1, 2], EPaxos exploits the commutativity between commands submitted to the replicated service to improve performance. In the most favorable case, that is when there is no concurrent non-commuting command, the protocol decides (aka., commit) the next command to execute after one round-trip to the closest fast quorum.

With more details, processes running the EPaxos protocol agree on a directed graph, or execution graph. Commands are executed on the local copy of the service following some linearization of this graph.111 Cycles in the execution graph are broken deterministically.

To build the execution graph, for each command $c$ submitted to the system, EPaxos constructs a set of dependencies $\mathit{dep}(c)$. The dependencies of $c$ precede it in the execution graph. Command $c$ is committed at some process $p$ once $\mathit{dep}(c)$ is known at $p$. A command is executed once the transitive closure of its dependencies is committed.

The EPaxos protocol satisfies two core invariants detailed below. These invariants ensure that the protocol maintains the consistency of the replicated service, namely its linearizability [7].

• (E1)

Any two processes agree on the dependencies of a command.

• (E2)

For any two non-commuting committed commands $c$ and $d$, c{\makebox[7.7778pt][c]{\in}}\mathit{dep}(d) or the converse holds.

To agree on the dependencies of a command, EPaxos employs a variation of Fast Paxos. The first (so-called) pre-accept phase of the agreement is not coordinated. More precisely, to propose some command $c$, a process $p$ propagates $c$ to a quorum. Upon receiving command $c$, a process sends all the commands conflicting with $c$ (that it is aware of) back to $p$. If all the processes return the same dependencies, a spontaneous agreement occurs. Otherwise, a classical accept phase takes place.

Let us notice that when ballot $b$ is not coordinated, processes may return different values for $\mathit{dep}(c)$, and invariant A4 does not hold anymore. When $b$ is fast, (A4) is replaced with the following Fast Paxos invariant [6]:

• (A4’)

For any two quorums $Q$ and $Q^{\prime}$ at ballot $b$, and any quorum $Q^{\prime\prime}$ at some ballot $b^{\prime}$, $Q\cap Q^{\prime}\cap Q^{\prime\prime}\neq\{\}$.

Invariant (A4’) ensures that a process starting ballot $b^{\prime}$ (a recovery in EPaxos parlance) observes at most a single chosen value at ballot $b$, the one with the most votes.

Differently from Fast Paxos, only ballot [math] is fast in EPaxos and allows a spontaneous agreement to form.222 During a recovery, the TryPreAccept messages serve solely to ensure invariant (E2).

In addition, when a process proposes a command, it includes its local computation of $\mathit{dep}(c)$ in its proposal. This allows to reduce the size of a fast quorum at ballot [math] to $f+\lfloor\frac{f+1}{2}\rfloor$ processes.

4 Breaking safety

As detailed in Section 2, consensus algorithms commonly employ two ballot variables at a process. These variables track the last joined ballot ($\mathit{bal}$) and the last ballot at which a value was voted ($\mathit{vbal}$).

However, both the ${\text{TLA}^{+}}$ specification of EPaxos [8, pages 109–123] and its implementation [9] use only $\mathit{bal}$.333 The specification and the implementation of EPaxos call ballot this unique variable.

When a process receives a message to join a ballot $b>\mathit{bal}$, it sends back $\mathit{bal}$ and $\mathit{vval}$, then updates $\mathit{bal}$ to $b$. This is surprising as the pseudo-code [3, Figure 3] seems to correctly identify the pattern, mentioning to send back “the most recent ballot number accepted” in the consensus instance.

Avoiding the use of a second ballot variable is not possible. Figure 1 illustrates how to break safety by executing a well-chosen sequence of steps.

In this figure, the system consists of three processes $\{p_{1},p_{2},p_{3}\}$. Processes $p_{3}$ and $p_{1}$ propose respectively to compute the dependencies of commands $c_{1}$ and $c_{2}$. This computation takes place respectively in the consensus instances $\langle p_{3},1\rangle$ and $\langle p_{1},1\rangle$. At the end of the execution, the value of $\mathit{dep}(c_{2})$ committed at process $p_{2}$ is $\{\}$, while it equals $\{c_{1}\}$ at processes $p_{1}$.

In appendix, we provide the ${\text{TLA}^{+}}$ code of the execution presented at Figure 1. This counter-example is located in a set of states which is too large to explore in a reasonable amount of time with the ${\text{TLA}^{+}}$ model checker. As a consequence, the specification in appendix directly injects the admissible execution with the help of a history variable. The model checker can be then used to validate that the counter-example is actually feasible. The history variable introduced in appendix is a counter. Its value coincides with the numbering of the steps in Figure 1.

The disagreement depicted in Figure 1 is obtained by executing consecutive recovery phases. It is based on the following observation: if $\mathit{vbal}$ is not used, a process that accepted a value $u$ at a ballot $b$, then later joins $b^{\prime}>b$ is in a state identical to having accepted $u$ at ballot $b$’.

In detail, the execution at Figure 1 consists of the following steps:

1. (1-2)

Process $p_{3}$ proposes command $c_{1}$, while $p_{1}$ proposes command $c_{2}$. 2. (3)

$p_{3}$ returns a pre-accept to $p_{1}$ message with $dep(c_{2})=\{c_{1}\}$. 3. (4-7)

$p_{3}$ partially recovers $\langle p_{1},1\rangle$ by contacting the quorum $\{p_{2},p_{3}\}$. This leads to the fact that $dep(c_{2})=\{c_{1}\}$ is accepted at ballot $2$ (step 7). 4. (8-14)

$p_{2}$ executes a full recovery of $c_{2}$ using $\{p_{1},p_{2}\}$. Before step 15, $dep(c_{2})=\{\}$ is thus accepted at process $p_{2}$. 5. (15-16)

$p_{1}$ executes a partial recovery with quorum $\{p_{1},p_{3}\}$. The state of instance $\langle p_{1},1\rangle$ at process $p_{3}$ is now $("{\textsf{accepted}}",3,\{c_{1}\})$. 6. (17-22)

$p_{1}$ executes a full recovery using the same quorum. As $p_{3}$ holds the highest ballot, its value for $\mathit{dep}(c_{2})$ is accepted at ballot $4$. 7. (23-24)

Processes $p_{1}$ and $p_{2}$ commit respectively $\{c_{1}\}$ and $\{\}$ for $\mathit{dep}(c_{2})$.

5 Conclusion

Egalitarian Paxos (EPaxos) is a recent protocol to implement state-machine replication and construct fault-tolerant distributed services. In the common case, EPaxos delivers a command after one round-trip to the closest fast quorum. Contrary to prior works, such as Generalized Paxos [2], a leader does not need to solve conflicts between non-commuting commands. These two properties make the protocol particularly appealing for geo-distributed systems.

The repeated consensus procedure of EPaxos must rely on two ballot variables to track the progress at each replica. This paper shows that if this is not the case, as in the ${\text{TLA}^{+}}$ specification and the Golang implementation, breaking safety is possible.

Acknowledgments

This research is partly supported by the RainbowFS project of Agence Nationale de la Recherche, France, number ANR-16-CE25-0013-01a, and the European Union’s Horizon 2020 research and innovation programme under grant agreement No 825184 (CloudButton). The author thanks Vitor Enes for his remarks on an initial version of this work, and Alexey Gotsman for fruitful discussions on the Egalitarian Paxos protocol.

Appendix A The counter example

A.1 ${\text{TLA}^{+}}$ specification

$\mbox{}{\textsc{module }}CounterExample\mbox{}$

$\mbox{}{\textsc{extends }}EgalitarianPaxos,\,TLC\mbox{}$

$\mbox{}{\textsc{constants }}p1,\,p2,\,p3,\,c1,\,c2\mbox{}$

$\mbox{}{\textsc{variables }}HIndex\mbox{}$

$\mbox{}MCReplicas\mbox{}\;\mathrel{\smash{{\stackrel{{\scriptstyle\scriptscriptstyle\Delta}}{{=}}}}}\;\mbox{}\hskip 4.1pt\{p1,\,p2,\,p3\}\mbox{}$

$\mbox{}MCCommands\mbox{}\;\mathrel{\smash{{\stackrel{{\scriptstyle\scriptscriptstyle\Delta}}{{=}}}}}\;\mbox{}\{c1,\,c2\}\mbox{}$

$\mbox{}MCFastQuorums(X)\mbox{}\;\mathrel{\smash{{\stackrel{{\scriptstyle\scriptscriptstyle\Delta}}{{=}}}}}\;\mbox{}{\textsc{if }}X\mbox{}=\mbox{}p1\mbox{}\textsc{ then }\mbox{}\{\{p1,\,p3\}\}\mbox{}$

$\mbox{}\mbox{}\makebox[26.11115pt][l]{{ else}}\mbox{}{\textsc{if }}X\mbox{}=\mbox{}p2\mbox{}\textsc{ then }\mbox{}\{\{p1,\,p2\}\}\mbox{}$

$\mbox{}\mbox{}\makebox[26.11115pt][l]{{ else}}\mbox{}\{\{p2,\,p3\}\}\mbox{}$

$\mbox{}MCSlowQuorums(X)\mbox{}\;\mathrel{\smash{{\stackrel{{\scriptstyle\scriptscriptstyle\Delta}}{{=}}}}}\;\mbox{}MCFastQuorums(X)\mbox{}$

$\mbox{}MCMaxBallot\mbox{}\;\mathrel{\smash{{\stackrel{{\scriptstyle\scriptscriptstyle\Delta}}{{=}}}}}\;\mbox{}5\mbox{}$

$\mbox{}AdvanceHistory(pos)\mbox{}\;\mathrel{\smash{{\stackrel{{\scriptstyle\scriptscriptstyle\Delta}}{{=}}}}}\;\mbox{}\mbox{}\land\mbox{}HIndex\mbox{}=\mbox{}pos\mbox{}$

$\mbox{}\mbox{}\land\mbox{}HIndex\mbox{}^{\prime}\mbox{}\mbox{}=\mbox{}HIndex\mbox{}+\mbox{}1\mbox{}$

$\mbox{}NewInit\mbox{}\;\mathrel{\smash{{\stackrel{{\scriptstyle\scriptscriptstyle\Delta}}{{=}}}}}\;\mbox{}\mbox{}$

$\mbox{}\hskip 8.2pt\mbox{}\land\mbox{}HIndex\mbox{}=\mbox{}1\mbox{}$

$\mbox{}\hskip 8.2pt\mbox{}\land\mbox{}sentMsg\mbox{}=\mbox{}\{\}\mbox{}$

\mbox{}\hskip 8.2pt\mbox{}\land\mbox{}cmdLog\mbox{}=\mbox{}[r\mbox{}{\makebox[7.7778pt][c]{\in}}\mbox{}Replicas\mbox{}\mapsto\mbox{}\{\}]\mbox{}

$\mbox{}\hskip 8.2pt\mbox{}\land\mbox{}proposed\mbox{}=\mbox{}\{\}\mbox{}$

\mbox{}\hskip 8.2pt\mbox{}\land\mbox{}executed\mbox{}=\mbox{}[r\mbox{}{\makebox[7.7778pt][c]{\in}}\mbox{}Replicas\mbox{}\mapsto\mbox{}\{\}]\mbox{}

\mbox{}\hskip 8.2pt\mbox{}\land\mbox{}crtInst\mbox{}=\mbox{}[r\mbox{}{\makebox[7.7778pt][c]{\in}}\mbox{}Replicas\mbox{}\mapsto\mbox{}1]\mbox{}

\mbox{}\hskip 8.2pt\mbox{}\land\mbox{}leaderOfInst\mbox{}=\mbox{}[r\mbox{}{\makebox[7.7778pt][c]{\in}}\mbox{}Replicas\mbox{}\mapsto\mbox{}\{\}]\mbox{}

\mbox{}\hskip 8.2pt\mbox{}\land\mbox{}committed\mbox{}=\mbox{}[i\mbox{}{\makebox[7.7778pt][c]{\in}}\mbox{}Instances\mbox{}\mapsto\mbox{}\{\}]\mbox{}

$\mbox{}\hskip 8.2pt\mbox{}\land\mbox{}ballots\mbox{}=\mbox{}1\mbox{}$

\mbox{}\hskip 8.2pt\mbox{}\land\mbox{}preparing\mbox{}=\mbox{}[r\mbox{}{\makebox[7.7778pt][c]{\in}}\mbox{}Replicas\mbox{}\mapsto\mbox{}\{\}]\mbox{}

$\mbox{}NewNext\mbox{}\;\mathrel{\smash{{\stackrel{{\scriptstyle\scriptscriptstyle\Delta}}{{=}}}}}\;\mbox{}\mbox{}\lor\mbox{}(AdvanceHistory(1)\mbox{}\land\mbox{}Propose(c1,\,p3))\mbox{}$

$\mbox{}\mbox{}\lor\mbox{}(AdvanceHistory(2)\mbox{}\land\mbox{}Propose(c2,\,p1))\mbox{}$

$\mbox{}\mbox{}\lor\mbox{}(AdvanceHistory(3)\mbox{}\land\mbox{}Phase1Reply(p3))\mbox{}$

$\mbox{}\mbox{}\lor\mbox{}(AdvanceHistory(4)\mbox{}\land\mbox{}SendPrepare(p3,\,{\langle}p1,\,1{\rangle},\,\{p2,\,p3\}))\mbox{}$

$\mbox{}\mbox{}\lor\mbox{}(AdvanceHistory(5)\mbox{}\land\mbox{}ReplyPrepare(p2))\mbox{}$

$\mbox{}\mbox{}\lor\mbox{}(AdvanceHistory(6)\mbox{}\land\mbox{}ReplyPrepare(p3))\mbox{}$

$\mbox{}\mbox{}\lor\mbox{}(AdvanceHistory(7)\mbox{}\land\mbox{}PrepareFinalize(p3,\,{\langle}p1,\,1{\rangle},\,\{p2,\,p3\}))\mbox{}$

$\mbox{}\mbox{}\lor\mbox{}(AdvanceHistory(8)\mbox{}\land\mbox{}SendPrepare(p2,\,{\langle}p1,\,1{\rangle},\,\{p1,\,p2\}))\mbox{}$

$\mbox{}\mbox{}\lor\mbox{}(AdvanceHistory(9)\mbox{}\land\mbox{}ReplyPrepare(p1))\mbox{}$

$\mbox{}\mbox{}\lor\mbox{}(AdvanceHistory(10)\mbox{}\land\mbox{}ReplyPrepare(p2))\mbox{}$

$\mbox{}\mbox{}\lor\mbox{}(AdvanceHistory(11)\mbox{}\land\mbox{}PrepareFinalize(p2,\,{\langle}p1,\,1{\rangle},\,\{p1,\,p2\}))\mbox{}$

$\mbox{}\mbox{}\lor\mbox{}(AdvanceHistory(12)\mbox{}\land\mbox{}Phase1Reply(p1))\mbox{}$

$\mbox{}\mbox{}\lor\mbox{}(AdvanceHistory(13)\mbox{}\land\mbox{}Phase1Slow(p2,\,{\langle}p1,\,1{\rangle},\,\{p1,\,p2\}))\mbox{}$

$\mbox{}\mbox{}\lor\mbox{}(AdvanceHistory(14)\mbox{}\land\mbox{}Phase2Reply(p1))\mbox{}$

$\mbox{}\mbox{}\lor\mbox{}(AdvanceHistory(15)\mbox{}\land\mbox{}SendPrepare(p1,\,{\langle}p1,\,1{\rangle},\,\{p1,\,p3\}))\mbox{}$

$\mbox{}\mbox{}\lor\mbox{}(AdvanceHistory(16)\mbox{}\land\mbox{}ReplyPrepare(p3))\mbox{}$

$\mbox{}\mbox{}\lor\mbox{}(AdvanceHistory(17)\mbox{}\land\mbox{}SendPrepare(p1,\,{\langle}p1,\,1{\rangle},\,\{p1,\,p3\}))\mbox{}$

$\mbox{}\mbox{}\lor\mbox{}(AdvanceHistory(18)\mbox{}\land\mbox{}ReplyPrepare(p3))\mbox{}$

$\mbox{}\mbox{}\lor\mbox{}(AdvanceHistory(19)\mbox{}\land\mbox{}ReplyPrepare(p1))\mbox{}$

$\mbox{}\mbox{}\lor\mbox{}(AdvanceHistory(20)\mbox{}\land\mbox{}ReplyPrepare(p1))\mbox{}$  answer both ballots

$\mbox{}\mbox{}\lor\mbox{}(AdvanceHistory(21)\mbox{}\land\mbox{}PrepareFinalize(p1,\,{\langle}p1,\,1{\rangle},\,\{p1,\,p3\}))\mbox{}$

$\mbox{}\mbox{}\lor\mbox{}(AdvanceHistory(22)\mbox{}\land\mbox{}Phase2Reply(p3))\mbox{}$

$\mbox{}\mbox{}\lor\mbox{}(AdvanceHistory(23)\mbox{}\land\mbox{}Phase2Finalize(p2,\,{\langle}p1,\,1{\rangle},\,\{p1,\,p2\}))\mbox{}$

$\mbox{}\mbox{}\lor\mbox{}(AdvanceHistory(24)\mbox{}\land\mbox{}Phase2Finalize(p1,\,{\langle}p1,\,1{\rangle},\,\{p1,\,p3\}))\mbox{}$

A.2 Model