Window Based BFT Blockchain Consensus

TL;DR

This paper introduces Musch, a BFT blockchain protocol that significantly reduces communication complexity to linear levels while maintaining low latency, addressing scalability issues without sacrificing fault tolerance.

Contribution

Musch is the first BFT blockchain protocol to achieve linear communication complexity with low latency under failures by using exponentially increasing windows for reporting complaints.

Findings

Achieves $O(f n + n)$ communication complexity for $f < n/3$ failures.

Maintains low latency despite reduced communication overhead.

Addresses scalability limitations of traditional BFT protocols.

Abstract

There is surge of interest to the blockchain technology not only in the scientific community but in the business community as well. Proof of Work (PoW) and Byzantine Fault Tolerant (BFT) are the two main classes of consensus protocols that are used in the blockchain consensus layer. PoW is highly scalable but very slow with about 7 (transactions/second) performance. BFT based protocols are highly efficient but their scalability are limited to only tens of nodes. One of the main reasons for the BFT limitation is the quadratic $O(n^2)$ communication complexity of BFT based protocols for $n$ nodes that requires $n \times n$ broadcasting. In this paper, we present the {\em Musch} protocol which is BFT based and provides communication complexity $O(f n + n)$ for $f$ failures and $n$ nodes, where $f < n/3$, without compromising the latency. Hence, the performance adjusts to $f$ such that for constant $f$ the communication complexity is linear. Musch achieves this by introducing the notion of exponentially increasing windows of nodes to which complains are reported, instead of broadcasting to all the nodes. To our knowledge, this is the first BFT-based blockchain protocol which efficiently addresses simultaneously the issues of communication complexity and latency under the presence of failures.

Tables

Table 1. TABLE I: Characteristics of state of Art BFT protocols. The actual faulty nodes is f 𝑓 f , while f ′ superscript 𝑓 ′ f^{\prime} is an upper bound, f ≤ f ′ 𝑓 superscript 𝑓 ′ f\leq f^{\prime} .

	PBFT	FastBFT	Aliph	Musch
Total Replicas $n$	$3f^{\prime }+1$	$2f^{\prime }+1$	$3f^{\prime }+1$	$3f^{\prime }+1$
Critical Path	4	$3+\log \left(f^{\prime }+1\right)$	$3f^{\prime }+2$	4
Communication Complexity	$O\left(n^2\right)$	$O\left(n^2\right)$	$O\left(n^2\right)$	$O\left(fn+n\right)$

Equations

$$\begin{array}{r}(2f+1)|R_c|+2(f+1)n+(f+1)|R_c|\\ \le (5f+4)n=O(fn+n).\end{array}$$

$$\begin{array}{r}2(2f^{\prime }+1)|R_c|\le (4f^{\prime }+2)f\\ <(4n/3+2)f=O(fn+n).\end{array}$$

$$\begin{array}{r}2(2f^{\prime }+1)|R_c|+(2f+1)n\\ <(4n/3+2)f+(2f+1)n=O(fn+n).\end{array}$$

$$(f+1)n+n+n=fn+3n.$$

$$2e(2f^{\prime }+f^{\prime })=6e{f}^{\prime }.$$

$$fn+3n+6e(n/3)=O(fn+n).$$

Full text

©2018 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works. DOI: [ 10.1109/Cybermatics_2018.2018.00184](<%20DOI:%2010.1109/Cybermatics_2018.2018.00184)

I Introduction

Consensus is used to agree on a new block to be appended to the chain by the nodes in the network. A blockchain is compromised of two main components:a cryptographic engine and a consensus engine. The main performance and scalability bottleneck of a blockchain also lies in these components. Here we only focus on improving consensus component of blockchains. As already mentioned, PoW-based protocols are highly scalable. In Bitcoin [1], which is one of the most successful implementation of blockchain technology, typically the number of nodes (replicas) are usually large in the range of thousands [1, 2].

PoW involves the calculation of a number based on the hash value of a block adjusted by a difficulty level. Solving this cryptographic puzzle by nodes (miners) limits the rate of the block generation as solving the puzzle is CPU intensive. Bitcoin uses PoW but the number of transactions per second can reach up to just 7 transactions per second [2]. The block generation rate is approximately 10 minutes [1]. Additionally, the power utilized by Bitcoin mining in 2014 was between 0.1-10 GW and was comparable to Ireland’s electricity consumption at that time [3]. Different solutions were proposed, for example, Ethereum [4] uses faster PoW, BitcoinNG [5] uses two types of blocks, namely, key blocks and micro-blocks, and has achieved $10\times$ more throughput in comparison with Bitcoin. But all these solutions fall well short of matching the throughput offered by leading credit-card companies ($2000$ on average and $10000$ maximum transactions per second).

On other the hand, BFT-based [6] protocols guarantee consensus in the presence of malicious (Byzantine) nodes, which can fail in arbitrary ways including crashes, software bugs and even coordinated malicious attacks. Typically, BFT-based algorithms execute in epochs, where in each epoch the correct (non-malicious) nodes achieve agreement for a set of proposed transactions. In each epoch there is a primary node that helps to reach agreement. The consensus is achieved during each epoch and an entry or a set of entries are added to the log. In case the primary is found to be Byzantine,a view change (select new primary) takes effect to provide liveness. These protocols have shown the ability to achieve throughput of tens of thousand transactions per second [7, 8]. However, their scalability has been tested with a very small number of nodes $n$, usually 10 to 20 nodes, due to the requirement for $n\times n$ broadcast [2], that is, they have quadratic communication complexity.

To address the scalability issues in BFT protocols, we introduce the Musch blockchain protocol. Musch is BFT-based and achieves $O(fn+n)$ communication complexity in an epoch, where $f$ is the actual number of Byzantine nodes ($f<n/3$). For small (i.e. constant) $f$ the communication complexity is linear, and hence, Musch has scalable performance. Musch does not need to know the actual value of $f$ since it automatically adjusts to the actual number of nodes that exhibit faulty behavior in each epoch. At the same time, the latency is comparable with other efficient BFT-based protocols [7, 9].

The performance of our algorithm is based on a novel mechanism of communication with a set of window nodes. Nodes reach sliding windows moving over node IDs to recover from faults during consensus. If a replica does not receive expected messages from the primary, it complains to the window nodes from which it recovers updates (see Fig. 1). Initially, the window consists of only one node. If the complainer replica doesn’t receive a valid response from any of the the window nodes, it considers the next window of double size to which it sends the complaint. The last window size is no more than $2f$ which guarantees to have a correct node within the last window. This gives $O(fn+n)$ communication complexity. In this way, Musch avoids $n\times n$ broadcasts while guaranteeing consistency.

Table I compares Musch with other state of art BFT-based protocols such as PBFT [9], FastBFT [10], and Aliph [8]. We compared the communication complexity measured as number of total exchanged messages during an epoch. Our algorithm’s performance depends solely on $fn$, while the other algorithms have quadratic communication complexity. Hence, our algorithm has an advantage when $f$ is asymptotically smaller than $n$, resulting in less than quadratic communication complexity. When $f$ is a constant our algorithm is optimal. Additionally, we also compared the critical path length, as the number of one-way message latency it takes for a client request to be processed and the response is received by the client. Note that the total number of nodes is $n=3f^{\prime}+1$, where $f^{\prime}$ is a conservative upper bound on the number of faulty nodes. The actual faults are bounded by $f\leq f^{\prime}$. Our algorithm does not need to know $f$.

SBFT [11] has also tried to address the issue of scalability and have tested their protocol with $100$ replicas, while achieving $10\times$ better performance than Ethereum. In normal mode when $f=0$, SBFT’s message complexity will be $O(nc)$ ($c$ is the number of collectors) as compared to Musch’s $O(n)$. The actual value of Byzantine nodes ($f$) has to be known (to choose correct $c$ such that $c\geq f$) for the system to avoid the fall-back protocol. But in practice it is impossible to know the actual value of $f$(avoiding fallback is not possible for small $c$). Fall-back mode executes efficient PBFT with $O(n^{2})$ complexity. This $O(n^{2})$ complexity causes additional latency and performance degradation in SBFT.

Paper Outline

We continue with this paper as follows. In Section II, we give the model of the distributed system. In Section III, we present our algorithm. Protocol checkpoints are presented in Section IV. We give the correctness analysis in Section V, and the communication complexity bound analysis in Section VI.

II System Model

Like other BFT-based state machine replication protocols Musch also assumes an adversarial failure model. Under this model, servers and even clients may deviate from their normal behavior in arbitrary ways, which includes hardware failures, software bugs, or malicious intent. Our protocol can tolerate up to $f^{\prime}$ number of Byzantine replicas where the total number of replicas in the network $n=3f^{\prime}+1$. Replica ID is an integer from the replica set $\{1,\ldots,n\}$ that identifies each replica. The actual number of Byzantine replicas in the network is denoted by $f$, and at any moment during execution $0\leq f\leq f^{\prime}$. If $f=0$ then the execution is fault-free. However, $f$ may not be known. Our algorithm’s communication complexity adapts to any value of $f$.

III Protocol

Our proposed protocol uses echo broadcast [12], where the primary proposes a block of transactions, and replicas respond by sending back signed hashes of the block. We assume strong adversarial coordinated attacks by various malicious replicas. However, replicas will not be able to break collision resistant hashes, encryption, or signatures. We assume that all messages sent by replicas and the primary are signed. For example if primary $p$ proposes a block of transactions $\langle B\rangle_{p}$ to the replica $i$, we assume that it has been signed by primary $p$. Any unsigned message will be discarded. To avoid repetition of message and signatures, Musch also uses signature aggregation [13] to use a single collective signature instead of appending all replica signatures, to keep signature size constant. As the primary $p$ receives message $M_{i}$ with their respective signatures $\sigma_{i}$ from each replica $i$, the primary then uses these received signatures to generate an aggregated signature $\sigma$. The aggregated signature can be verified by replicas given the messages $M_{1},M_{2},\ldots,M_{y}$ where $y\leq n$, the aggregated signature $\sigma$, and public keys $PK_{1},PK_{2},\ldots,PK_{y}$. Like other BFT-based protocols [9, 14, 15] each replica $i$ knows the public keys of other replicas in the network. In Section III-B we explain how to use the IDs to define the windows that we use in the algorithm.

It is not possible to ensure the safety and liveness of consensus algorithms in asynchronous systems where even a single replica can crash fail [16]. Musch’s safety holds in asynchronous environments. But to circumvent this impossibility for liveness, Musch assumes partial synchrony [17]. This partial synchrony is achieved by using arbitrarily large unknown but fixed worst case global stabilization delays.

During normal operation, Musch guarantees that at least $2f^{\prime}+1$ replicas in each epoch are consistent (out of the $n=3f^{\prime}+1$). Let $T$ be the maximum round-trip message delay in the network. In our algorithm, at any moment of time, the suffix of the execution histories between any two replicas differ by at most the maximum number of blocks that can be committed during a time period of $O(T\log f^{\prime})$. Thus, any inconsistency is limited to only a small period of time.

Musch executes in epochs. An epoch is a slot of time in which $2f^{\prime}+1$ replicas receive block $B$ proposed by the primary $p$ and agree to commit it. Thus, during each epoch a block is generated and added to the chain. Since $p$ is responsible for aggregating replica signatures for block agreement, if less than $2f^{\prime}+1$ replica signatures are collected then a view change will be triggered and the primary will be changed. It should be noted that $p$ is also responsible for collecting transactions from clients, ordering the transactions, and sending them to the replicas.

III-A Normal Operation

As shown in Algorithm 1, the primary $p$ collects a set of transactions from the clients into an ordered list of transactions $L^{a}$ (which it will propose in a candidate block) with a sequence number $s$, view number $v$, hash $d=hash(L^{a})$, and hash history $h_{s}=Hash(h_{s-1},d)$ into candidate block $B=\langle\langle ORDER,s,v,d,h_{s}\rangle_{p},L^{a}\rangle$. Primary $p$ then proposes (broadcasts) the candidate block $B$ to each replica $i$. As shown in Algorithm 2, upon receipt of $B$ each replica $i$ validates the information, and then replica $i$ responds to the primary $p$ with the willingness to accept the block in a message $H_{i}=\langle RESPONSE,s,v,d,i\rangle_{i}$ to the primary.

The primary collects at least $2f^{\prime}+1$ responses from the replicas, aggregates them to $H$, and generates a compressed aggregated signature $\sigma$ [13]. Then, the primary broadcasts $\langle COMMIT,H\rangle_{\sigma}$. Upon receipt, each replica $i$ verifies $2f^{\prime}+1$ signatures and the candidate block $B$ commits. If verified successfully each replica $i$ responds to the client with the reply message $\langle REPLY,s,v,c,r,t,i\rangle_{\sigma_{i}}$, where $c$ is the client, $t$ is the timestamp and $r$ is the result of execution. Upon receipt of $f^{\prime}+1$ valid $REPLY$ messages (which might take $2f^{\prime}+1$ messages to receive) a client accepts the result. Assuming a continuous creation of blocks, the primary starts the new epoch immediately after the old epoch finishes. Let $T$ be the maximum delivery delay of a message in the network. According to the protocol, in the epoch of the new block $B$ with sequence number $s$ there will be two messages that replica $i$ expects to receive from the primary: (i) the $ORDER$ type message for block $B$ with sequence $s$ within $\Delta_{2}=T$ time from the end of the previous epoch, and then (ii) the $COMMIT$ type message for block $s$ within $\Delta_{3}=2T$ time since the receipt of the $ORDER$ message. Therefore, the maximum time for an epoch for a replica $i$ is $\Delta_{1}=\Delta_{2}+\Delta_{3}$. A replica $i$ goes into recovery mode at time $\Delta_{1}$ if either of the two expected messages is not received.

III-B Recovery Mode

In BFT protocols, when a replica detects an error it broadcasts complaints to all replicas in the network. In contrast to this, a replica in Musch during a failure event will only complain to a subset of replicas in the network called window nodes. If $i$ did not receive a response from the current window then the replica complains to the next window of double size until it receive response from at least one correct replica. The window sequences are fixed, $W_{1},W_{2},\ldots,W_{k^{\prime}}$, where $k^{\prime}=\lceil\lg(f^{\prime}+1)\rceil$. Suppose the replica IDs are taken from the set $\{1,\ldots,n\}$ and sorted in ascending order (see Fig. 1). The window $W_{1}$ consists of a single node with the smallest ID, Window $W_{2}$ consists of two replicas with the next IDs in order, Window $W_{3}$ consists of four replicas with the next higher IDs, and so on. Therefore the window $W_{j}$ consists of $2^{j-1}$ replicas, whose IDs are ranked between $2^{j-1},\ldots,2^{j}-1$. During the execution of the algorithm, the maximum window that will be contacted is actually $k=\lceil\lg(f+1)\rceil$, where $k\leq k^{\prime}$, since this guarantees that at least one correct node will be encountered among all the window nodes from $W_{1}$ up to $W_{k}$.

Algorithm 3 describes how a replica complains to the window(s), and Algorithm 4 shows the respective reactions from the window nodes. As shown in Algorithm 3, if replica $i$ complains that it didn’t receive expected message ($ORDER$ or $COMMIT$) from $p$ during normal operation, it sends the complaint in the form of $\langle COMPLAIN,s,v,d\rangle_{i}$, where $d$ and $s$ belongs to the last committed block in the chain of $i$. If it complains to a window $W_{j}$, this message is sent to all nodes in $W_{j}$ which will then know that replica $i$ does not have $ORDER$ or $COMMIT$ messages after block $s$. If replica $i$ has received a message from the primary that proves the maliciousness of $p$, then it attaches the proof in its complaint $\langle COMPLAIN,PROOF\rangle_{i}$ to $W_{j}$.

When $i$ enters the recovery mode it first complains to window $W_{1}$, which has a single node. If $i$ doesn’t get any useful response from $W_{1}$ then it complains to $W_{2}$, which has two nodes, so it informs both nodes. This process can repeat until $i$ contacts all nodes in $W_{k}$, the last window. It is guaranteed that replica $i$ will get a response from a correct node in one of these windows. As shown in Algorithm 4, the window nodes respond to complaints by returning the requested information. If they do not have it then they call themselves Algorithm 3 as well. If the complainer $i$ is a window node itself, it will stop until it reaches its own window size and will broadcast the complaint. Upon broadcast it is guaranteed that it will receive response. The response can be either receipt of missing messages or a view change. If replica $i$ received the missing messages it will forward it to the complainers that it knows, else it will result in view change (primary will be replaced).

Note that regular replicas and window nodes may be complaining at the same time and probably for the same reason. A regular node will have to wait for the window nodes to first obtain a response. It is important to coordinate the actions of the windows nodes and the regular replicas to receive the responses efficiently without message replication. For a regular replica $i$, the timeout period for waiting a response from the window $W_{j}$ is at most $\Lambda_{j}=j3T+6T$. As it takes $\Delta_{1}=3T$ to detect timeout for the current epoch, then it takes at most $j3T$ to receive a message from the previous window and send the message back to the replica. In case window $j$ does not receive a message from window $j-1$, it will broadcast its complaint and it is guaranteed that it will receive a response, which it will send back to replica $i$ ($3T$). From the start time $t$ of the current epoch, if $i$ does not get a response within $t+\Lambda_{j}$ then it will contact the next window $W_{j+1}$.

III-C View Change

A view change can be triggered if a correct window node $i\in W_{l}$ receives at least $f^{\prime}+1$ distinct replica complaints (against primary $p$) as shown in Algorithm 4. This guarantees that at least one of the complaints is coming from a correct replica.

Another reason for view change can be the receipt of an explicit $PROOF$ against $p$ by window node $i$. Once view change is triggered, window node $i$ broadcasts the set of $COMPLAIN$ or $PROOF$ messages it has received to all replicas (Algorithm 4).

Without loss of generality, consider the case where window node $i$ has sent $PROOF$ to all replicas (the same mechanism also applies to other sets of $COMPLAIN$ messages). Upon receipt of $PROOF$ a replica $j$ increments its view number ($v=v+1$) and assigns new primary $p^{\prime}$ (namely, $p^{\prime}=v\mod n$) (Algorithm 5).

Replica $j$ then adds its most recent block hash $d$ and block number $s$ in the message along with $PROOF$ in a message $\langle VIEWCHANGE,PROOF,s,d,j\rangle_{j}$ and sends it to the new primary $p^{\prime}$ (Algorithm 5).

Upon receipt of at least $2f^{\prime}+1$ view change messages from different replicas, $p^{\prime}$ stores them into set $Q$. Then, $p^{\prime}$ broadcasts $\langle Q\rangle_{\sigma}$, where $\sigma$ is an aggregated signature for all replicas involved in $Q$ (Algorithm 6). Upon receipt of this message, each replica recovers the latest block history. Assume $s^{\prime}$ is the highest block number committed so far in the chain. The block $s^{\prime}$ must have been committed by at least $2f^{\prime}+1$ replicas, and since $Q$ has size at least $2f^{\prime}+1$, it must be that $f^{\prime}+1$ replicas in $Q$ have also committed $s^{\prime}$, one of which is a correct node. Thus, every replica upon receipt of $Q$ can figure out that the latest committed valid block number is $s^{\prime}$.

Once $s^{\prime}$ is known, a replica $i$ will check if block with sequence $s^{\prime}$ is the latest block in its history $h_{i}$, and if it is, $i$ sends a confirmation message $s^{\prime}_{i}$ to $p^{\prime}$ (Algorithm 5).

In this case, at least $f^{\prime}+1$ correct replicas know the latest block of $p^{\prime}$ ($s^{\prime}_{p^{\prime}}$). If $s^{\prime}$ is same as $s^{\prime}_{p^{\prime}}$ then $p^{\prime}$ begins updating all other replicas that have fallen behind (Algorithm  6). $p^{\prime}$ will not send any block generated earlier than the water mark $H$ (Section IV). If $p^{\prime}$ does not have $s^{\prime}$ as its latest block then at least $f^{\prime}+1$ correct replicas know about it and they send missing blocks and their respective $COMMIT$ messages to $p^{\prime}$ and then $p^{\prime}$ updates other replicas as described above. Once $p^{\prime}$ has updated other replicas it will wait to receive at least $2f^{\prime}+1$ correct replicas have sent confirmation $\ s^{\prime}_{i}$ (Algorithm 6).

Since there are at least $2f^{\prime}+1$ correct replicas, $p^{\prime}$ signs the latest block in their histories that $p^{\prime}$ has received using an aggregated signature $V\leftarrow\bigcup_{i}\langle s^{\prime}_{i}\rangle_{\sigma}$ and broadcasts it to the replicas. Upon receipt of $V$ each replica is now ready for the new epoch of the next block and is waiting to receive an $ORDER$ message from the new primary $p^{\prime}$ (Algorithm 5). In case a replica does not receive expected messages ($Q$, $V$ or blocks and their hashes within a certain expected time), then it issues a new complaint which is processed similar to the other types of complaints as described above.

During the view change process there may be some clients who send their request but it will not be processed because replicas are busy. To address this as we mentioned earlier the client $c$ will broadcast its request after epoch time $\Delta_{1}$, if it did not receive the response from $p$. In such case, all replicas receive the request $T_{c}$ and forward it to the $p$. Upon receipt of $2f^{\prime}+1$ such forwarded requests, $p$ considers $T_{c}$ to be included in the $ORDER$ message as soon as possible. $p$ will have to propose those backlogged requests before proposing the new requests it receives. If it proposes a request that has not been seen by $2f^{\prime}+1$ replicas (of which $f^{\prime}+1$ replicas are correct/honest replicas) proposing the backlogged transactions then the replicas can send a complaint which will result in a view change.

IV Checkpoints

As an optimization to the protocol, we use checkpoints to improve on the number of messages exchanged during view change. Checkpoints are typically used as a way to truncate the log in other BFT-based protocols [9]. In addition to that, we can also use it to prevent malicious replicas from downloading older messages from a new primary $p^{\prime}$ and delaying the completion of the view change process. As we know from Section III-B, some correct replicas might miss messages and go into recovery mode. These replicas need to download those missing messages. But malicious replicas might try to download very old blocks and delay the view change process. To bound this we use checkpoints. To maintain the safety condition it is required that at least $2f^{\prime}+1$ replicas agree on the checkpoint. The checkpoint is created after a constant number of blocks (e.g., sequence number divisible by 200). In Musch, replicas can agree on checkpoints during block agreement (checkpoint number to be added to the $RESPONSE$ message). A checkpoint that is agreed upon by $2f^{\prime}+1$ replicas of which at least $f^{\prime}+1$ are honest is called a stable checkpoint. Checkpoints have low and high watermarks. Low watermark $h$ is the last stable checkpoint and the high water mark $H$ is the sum of low water mark and $k$ number of blocks($H=k+h$), where $k$ is large enough (i.e. $k=400$). If a replica wants to download a block older than $H$, $p^{\prime}$ will ignore the download request and might think that the replica is maliciously trying to delay the view change process.

V Correctness Analysis

In this section we provide proof of correctness and analysis of the Musch protocol. Before we proceed, it is important to define transaction completion and protocol correctness for the Musch protocol. We say that a transaction $T_{c}$ issued by a client $c$ is considered to be completed by $c$ if $c$ receives at least $f^{\prime}+1$ valid $\langle REPLY,s,v,c,r,t,i\rangle_{\sigma_{i}}$ messages. It is guaranteed that upon receipt of $2f^{\prime}+1$ $REPLY$ messages from different replicas at least $f^{\prime}+1$ of them are valid. We will prove that Musch satisfies the following correctness criteria:

Definition 1** (Liveness).**

Every transaction proposed by the correct client will eventually be completed in finite time.

Definition 2** (Safety).**

A system is safe if a correct primary proposes a block of ordered transactions with block number $s$ and it is committed by at least $2f^{\prime}+1$ replicas, then any block that has been committed earlier will have smaller block number ($s^{\prime}<s$) in the chain. Thus, block $B_{s^{\prime}}$ will be the prefix of block $B_{s}$ in the chain. Additionally the order of transactions within the block will remain identical in all correct replicas (due to Merkle tree111Merkle trees are hash-based data structures in which each leaf node is hash of a data block and each non leaf node is hash of its children. It is mainly used for efficient data verification.).

V-A Safety

Lemma 1**.**

Any two committed blocks $B_{s^{\prime}}$ and $B_{s}$ must have a different block number.

Proof.

Consider committed blocks $B_{s^{\prime}}$ and $B_{s}$. At least a set of $2f^{\prime}+1$ replicas $S_{1}$ have agreed to all transactions with $B_{s^{\prime}}$ and have committed it. Similarly, at least a set of $2f^{\prime}+1$ replicas have agreed for the transactions in block $B_{s}$ and committed it. Since there are $3f^{\prime}+1$ replicas, there is at least one correct replica (out of the at least $f^{\prime}+1$ replicas in $S_{1}\cap S_{2}$) that committed both for $B_{s^{\prime}}$ and $B_{s}$. But a correct replica only commits one block with a specific block number. Thus, both blocks must have different numbers. The same mechanism applies during recovery mode. ∎

Lemma 2**.**

If block $B_{s^{\prime}}$ commits earlier than block $B_{s}$, then $B_{s^{\prime}}$ has a smaller block number than $B_{s}$.

Proof.

As per Lemma 1, at least one correct replica $k$ has committed both $B_{s^{\prime}}$ and $B_{s}$. Suppose, that $B_{s^{\prime}}$ gets a block number $s^{\prime}$ which is smaller than the block number $s$ of $B_{s}$, that is $s^{\prime}<s$ ($s\neq s^{\prime}$ from Lemma 1). A correct Replica $k$ will only accept $B_{s}$ if $B_{s}$ is consistent with its local history (only if $s>s^{\prime}$ ). ∎

Lemma 3**.**

Musch is safe during view change.

Proof.

During a view change (Algorithms 5 and 6), all replicas including the new primary $p^{\prime}$ retrieve the latest history and block number $s^{\prime}$ as at least $f^{\prime}+1$ replicas will agree on the latest block number $s^{\prime}$, which includes a correct replica that knows $s^{\prime}$. All correct replicas know the latest block $s^{\prime}_{p^{\prime}}$ in the history of $p^{\prime}$ from $\langle Q\rangle_{\sigma}$. If $s^{\prime}_{p^{\prime}}=s^{\prime}$ then $p^{\prime}$ begins updating all other replicas that have fallen behind in history, in other words it updates all the replicas that do not have blocks and respective $COMMIT$ messages up to $s^{\prime}$. If $p^{\prime}$ does not have $s^{\prime}$ as its latest block then at least $f^{\prime}+1$ correct replicas know about it (from $Q$) and they send missing blocks and their respective $COMMIT$ messages to $p^{\prime}$ and then $p^{\prime}$ updates other replicas as described above. Once $p^{\prime}$ updated (receive blocks and $COMMIT$s up to $s^{\prime}$) other replicas it will take $T$ timeout period to receive $s_{i}$ (update confirmation) from at least $2f^{\prime}+1$ replicas. Then, $p^{\prime}$ signs all their histories using an aggregated signature $V\leftarrow\bigcup_{i}\langle s_{i}\rangle_{\sigma}$ and broadcasts it to the replicas. Upon receipt of $V$ each replica is now ready for the new epoch of the next block and is waiting to receive an $ORDER$ message from the new primary $p^{\prime}$. ∎

Theorem 4** (Safety).**

Musch is safe.

Proof.

Lemma 3 guarantees safety when the new primary $p^{\prime}$ is correct. If $p^{\prime}$ is not correct, safety will be guaranteed when eventually a correct primary will be chosen. Therefore, based on Lemmas 1, 2 and 3, Musch is safe when replicas are either in normal, recovery, or view change mode. ∎

V-B Liveness

In this section we provide a proof for liveness of Musch.

Lemma 5**.**

Musch satisfies liveness when the primary is correct.

Proof.

Consider a correct primary $p$ that executes Algorithm 1, and also the replicas that execute Algorithm 2. Primary $p$ receives at least $2f^{\prime}+1$ correct $RESPONSE$ messages from replicas, aggregates and signs them using an aggregation signature $\sigma$. It then broadcasts the signed $COMMIT$ message to all replicas. Upon receipt of the $COMMIT$ message each replica will commit the block. The primary $p$ along with all correct replicas also forwards a reply message to each client $\langle REPLY,s,v,c,r,t,i\rangle$ and clients will mark the transaction as completed. ∎

Lemma 6**.**

If there are $f^{\prime}+1$ complaints, or there is a complaint with a proof of maliciousness against the primary, then a view change will occur.

Proof.

Algorithm 3 guarantees that, in the worst case, a replica $i$ can find a window node $W_{k}$ to complain, where, $k=\lceil\lg(f+1)\rceil$ and $W_{k}$ contains at least one correct replica, since $W_{k}$ contains at least $2^{\lg(f+1)}=f+1$ nodes. Observe that once a replica $i$ has found a honest window node, it is guaranteed that the honest node will reply to its valid complaint either by sending back blocks and $COMMIT$s or if the number of complaints are greater than $f^{\prime}$, then the window node will broadcast all complaints to the network causing a view change (Algorithm 4).

If a replica $j\in W_{k}$ receives at least $f^{\prime}+1$ complaints from other replicas it triggers a view change according to the Algorithm 4. Since $f^{\prime}+1$ complaints are received, this guarantees that at least one honest replica has complained.

Similarly, $j$ may receive an explicit proof that the primary $p$ is faulty ($p$’s history is incorrect, or it has proposed an invalid transaction, etc.). In such a case only one complaint is needed to prove that $p$ is malicious and a view change will be triggered. ∎

Lemma 7**.**

If a transaction is not completed then a view change will occur.

Proof.

If a transaction does not complete after sufficient time $\Delta_{1}$, then the client $c$ broadcasts its transaction $T_{c}$ to the replicas. Upon receipt of $T_{c}$, the replicas check if they have already committed a block that contains $T_{c}$. If they did, each replica $i$ will send $\langle ACK,T_{c}\rangle_{i}$ to the client and upon receipt of $2f^{\prime}+1$ $ACK$ messages the client will consider the transaction as complete. If primary $p$ has not proposed the transaction $T_{c}$, then each replica will forward $T_{c}$ to $p$ and will expect that $p$ will include it in the next $ORDER$ message (during normal operation). If $p$ does not include it in the next $ORDER$ message, then replicas will start complaining, which will result in a view change (if at least $f^{\prime}+1$ replicas complain, from Lemma 6).

Another case that can prevent a request from being committed is when replicas receive a $COMMIT$ message signed by less than $2f^{\prime}+1$ replicas. In this case, this can be used as proof against $p$ and a complaint can be made, which will result in a view change (Lemma 6). ∎

Lemma 8**.**

Musch satisfies liveness even if a client request is received during a view change.

Proof.

During the view change process, there may be some clients who send their request for transaction $T_{c}$ but it will not be processed because replicas are busy with the view change. To address this, as mentioned earlier the client $c$ will broadcast its request after epoch timeout $\Delta_{1}$, if it did not receive a response from $p$. In such a case, all replicas receive the request $T_{c}$ and forward it to the new primary $p^{\prime}$. Upon receipt of $2f^{\prime}+1$ such forwarded requests the $p^{\prime}$ considers $T_{c}$ to be included in the $ORDER$ message as soon as possible. The new primary $p^{\prime}$ will have to propose those backlogged client requests during the view change, before proposing the new requests it receives. If it proposes a request that has not been seen by $2f^{\prime}+1$ replicas (of which $f^{\prime}+1$ replicas are correct/honest replicas), proposing the backlogged transactions then the replicas can start complaints, which will result in a new view change (Lemma 6). ∎

Theorem 9** (Liveness).**

Musch satisfies liveness and all correct transactions will be completed eventually.

Proof.

Based on Lemmas 5, 7 and 8, any correct transaction request by a client will be completed within a finite period of time. ∎

VI Communication Complexity

In communication complexity, we count all messages that cause a reaction in our algorithm and we refer to these as effective messages. In contrast, there are ineffective messages, which have sources that have been identified as malicious, and so the recipient can ignore these messages. We will measure the number of effective messages exchanged in an epoch, and we will consider worst cases scenarios, with or without view change. In other words, we consider worst-case performance attacks when malicious replicas attempt to increase the communication of the protocol by causing messages to be sent from correct replicas.

In the communication complexity we consider separately the messages sent between clients and replicas, and those sent only between replicas.

VI-A Client-Replica Communication Complexity

If a client sends a transaction to the primary $p$, and does not receive a response from the primary $p$ within $\Delta_{1}$, then the client broadcasts to the primary $p$ (a broadcast involves $n$ messages). Upon receipt of a broadcast from a client, if replica $i$ has already processed the client’s transaction it will answer to the client with an acknowledgement. If not, the replica $i$ will forward the client’s request to the primary, forcing it to process it as soon as possible. The liveness property of our algorithm, Theorem 9, will guarantee that eventually at least $2f^{\prime}+1=O(n)$ of the replicas will send acknowledgements to the client. Therefore, we get the following result:

Lemma 10**.**

For each transaction sent by a client, at most $O(n)$ messages will be exchanged between the client and the replicas in order to process the transaction (i.e., include the transaction in a block).

VI-B Replica-Replica Communication Complexity

In this section we analyze the communication complexity of the consensus engine of our protocol, which includes the primary $p$ and the replicas (in total $n$ nodes). A malicious primary $p$ and malicious replicas both can try to increase the communication complexity.

VI-B1 Messages caused by malicious primary

Let $R_{c}$ be the set of replicas that complain. First, we examine the case when the nodes in $R_{c}$ did not receive the block or $COMMIT$ message and they complain. A malicious primary $p$ can afford not to send such messages up to at most $f^{\prime}$ replicas, without getting caught as being malicious; that is, $|R_{c}|\leq f^{\prime}$.

In this case, each of the complainers in $R_{c}$ may have to communicate with up to $2f+1$ window nodes, since this guarantees a window that has at least one correct window node. This gives at most $(2f+1)|R_{c}|$ messages. In the worst case, out of the $2f+1$ window replicas at most $f+1$ will be the honest ones that will broadcast to all $n$ replicas and will receive their response, to be forwarded to the complainers $R_{c}$, giving at most $2(f+1)n+(f+1)|R_{c}|$ additional messages. The total communication complexity in this case will be (since $|R_{c}|\leq f^{\prime}<n/3$):

[TABLE]

VI-B2 Messages caused by malicious replicas

Suppose the set of complainers $R_{c}$ are malicious, thus, $|R_{c}|\leq f$. Window nodes do not respond to repetitive complains from the same replica (non-effective messages), which prevents malicious replicas from increasing the communication complexity. Nevertheless, each window node may respond once to each malicious request. A window node $j$ can respond to a complain message in the following ways:

• •

If window node $j$ has the appropriate response to the complain (i.e. it has the block or $COMMIT$) it will send it back to the replica that complained. At most $2f^{\prime}+1$ window nodes will be accessed by each replica in $R_{c}$, since this is the bound on the total number of window nodes. Therefore, in this case, the number of messages are at most:

[TABLE]

• •

If window node $j$ does not have the appropriate response (block or $COMMIT$), then $j$ itself is also executing the window protocol from smaller to larger windows, and when it eventually points to its own window, it will broadcast the complaint to get a response from other replicas (acting as a regular window node). This scenario can only happen if all the previous windows are populated by $f$ faulty nodes. The number of complaints from $R_{c}$ to up to $2f^{\prime}+1$ window nodes are bounded by $(2f^{\prime}+1)|R_{c}|$. Similarly, the respective responses are bounded by $(2f^{\prime}+1)|R_{c}|$. For calculating the messages from the broadcasts, out of the $2f^{\prime}+1$ total window nodes, at most $2f+1$ window nodes will react to the received complaints with broadcasts, since the first encountered window of size at least $f+1$ will respond to any complaint from a valid node. Thus, each of the up to $2f+1$ windows nodes broadcasts to all replicas, causing $(2f+1)n$ additional messages. Therefore, in this case, the number of messages are at most:

[TABLE]

VI-C View Change Communication Complexity

When a correct window node receives $f^{\prime}+1$ complaints it will broadcast all of them to all replicas ($n$ messages). There are at most $f+1$ window nodes that will broadcast (since those window nodes could be correct in the last accessed window size), resulting to at most $(f+1)n$ messages. Upon receipt of the broadcast message, each replica begins the view change process. The replica sends back a $VIEWCHANGE$ message to the new primary $p^{\prime}$ which also includes its history ($n$ messages). The new primary $p^{\prime}$ aggregates all $VIEWCHANGE$ messages into $\langle Q\rangle_{\sigma_{p}^{\prime}}$ and broadcasts ($n$ messages). Upon receipt each replica extracts the most recent block as described in Section III-C. Therefore, the number of messages from this part of the algorithm is at most:

[TABLE]

During this, at least $f^{\prime}+1$ correct replicas have the latest committed block $s^{\prime}$, and this block is chosen as the starting point for the next epoch, which will build another block ($s^{\prime}+1$) over it. All other replicas that have block number less than $s^{\prime}$ as their latest block have to download all the blocks up to $s^{\prime}$ from $p^{\prime}$. If $p^{\prime}$ does not have $s^{\prime}$ as its latest block, then $f^{\prime}+1$ replicas that have it will bring $p^{\prime}$ up to date. Thus, if $f^{\prime}+1$ replicas have $s^{\prime}$ as their latest block, then, at most $2f^{\prime}$ replicas in the worst case get (download) messages up from the high water mark in checkpoint $H$ to $s^{\prime}$. Let $e$ be the number of committed blocks from $H$ to $s^{\prime}$. For each committed block we need two messages, first the block itself and the second is the $COMMIT$ message. Thus, we have:

[TABLE]

Assuming frequent checkpoints (say every a fixed number of blocks), we can assume that $e$ is a constant. From Equations 4 and 5 we have for the total number of messages in view change:

[TABLE]

VI-D Overall Messages

Combining Equations 1, 2, 3, we obtain $O(fn+n)$ communication complexity in a single epoch for the communication complexity between replicas. From Equation 6 the communication complexity is also $O(fn+n)$ during view change. Therefore, we have the following result:

Lemma 11**.**

The number of messages exchanged between replicas in an epoch or during view change are $O(fn+n)$.

Combining Lemmas 10 and 11 we obtain the main result for the communication complexity:

Theorem 12** (Communication complexity).**

For $\tau$ initiated transactions in an epoch, the communication complexity is $O(\tau n+fn)$. For constant $\tau$, the communication complexity is $O(fn+n)$.

VII Conclusions

In this paper we proposed Musch, a BFT-based consensus protocol, in an effort to avoid excessive messages and improve the scalability of blockchain algorithms. Through the use of windows, the algorithm adapts to the actual number of faulty nodes $f$, and in this way it avoids unnecessary messages. This improvement does not sacrifice on the latency, since our algorithm still uses a small number of communication rounds. For future work, it would be interesting to investigate whether we can decrease the message complexity further, i.e. to $O(n)$ under $f$ faults, by introducing an intelligent scheme to detect faulty nodes and foil attempts to increase message complexity.