Ostraka: Secure Blockchain Scaling by Node Sharding

TL;DR

Ostraka introduces a node sharding architecture for blockchains that enhances scalability and security by parallelizing node processing, avoiding overload attacks, and maintaining consensus integrity.

Contribution

This paper presents Ostraka, a novel node sharding approach that improves blockchain scalability and security without requiring costly coordination among nodes.

Findings

Ostraka nodes' transaction processing rate scales linearly with resources.

Ostraka prevents denial-of-service attacks by distributing transaction load.

Replacing a unified node with Ostraka does not compromise security.

Abstract

Cryptocurrencies, implemented with blockchain protocols, promise to become a global payment system if they can overcome performance limitations. Rapidly advancing architectures improve on latency and throughput, but most require all participating servers to process all transactions. Several recent works propose to shard the system, such that each machine would only process a subset of the transactions. However, we identify a denial-of-service attack that is exposed by these solutions - an attacker can generate transactions that would overload a single shard, thus delaying processing in the entire system. Moreover, we show that in common scenarios, these protocols require most node operators to process almost all blockchain transactions. We present Ostraka, a blockchain node architecture that shards (parallelizes) the nodes themselves. We prove that replacing a unified node with an Ostraka node does not affect the security of the underlying consensus mechanism. We evaluate analytically and experimentally block propagation and processing in various settings. Ostraka allows nodes in the network to scale, without costly coordination. In our experiments, Ostraka nodes' transaction processing rate grows linearly with the addition of resources.

Tables

Table 1. TABLE I: Ostraka compared to other systems

	OmniLedger	RapidChain	Ostraka
Security	25%	33%	50%
Cross-Shard Tx	By Clienta	By leaderb	Internal
Required Nodes	hundreds	hundreds	any
Throughput	10k tx/sc	7.3k tx/sd	10k tx/se
DoS resilience	✘	✘	✔

Equations

$$\text{Capacity}\left[\frac{\text{TX}}{\text{sec}}\right]=\frac{1}{\frac{\text{TxSize}}{\text{BW}}+\left(\frac{1}{{\text{NS}}_{\text{tps}}\cdot \text{\#NSs}}\right)}$$

$$Bpt=\frac{\text{BSize}}{\text{TxSize}}\cdot \frac{1}{\text{\#NSs}}\cdot \left(\frac{1}{{\text{NS}}_{\text{tps}}}+\frac{\text{\#NSs}-1}{\text{\#NSs}}\cdot \frac{{\text{TX}}_{out}}{{\text{BW}}_{\text{NS}}}\right)$$

Full text

However, we identify a denial-of-service attack that is exposed by these solutions – an attacker can generate transactions that would overload a single shard, thus delaying processing in the entire system. Moreover, we show that in common scenarios, these protocols require most node operators to process almost all blockchain transactions.

We present Ostraka, a blockchain node architecture that shards (parallelizes) the nodes themselves. We prove that replacing a unified node with an Ostraka node does not affect the security of the underlying consensus mechanism.

We evaluate analytically and experimentally block propagation and processing in various settings. Ostraka allows nodes in the network to scale, without costly coordination. In our experiments, Ostraka nodes transaction processing rate grows linearly with the addition of resources.

1 Introduction

Payment systems are typically controlled by trusted entities – central banks control supply by printing money, banks transact among each other with a central national [34] or international [33] mechanism, and credit card companies [36] authorize their clients’ transactions. This centralization of power implies limitations of the users’ freedom [31], less resilience, and high fees [5].

Since their introduction with Bitcoin [29], cryptocurrencies brought the promise of a decentralized global payment system outside the control of a single entity. Such a system is potentially useful for payment among individuals, but also among banks, machines (IoT), and micro-payments for social network interactions.

So far, blockchains systems – the decentralized mechanism on which cryptocurrencies operate – could not reach rates desired for global payment systems. Early system implementations suffered from a consensus bottleneck, limiting them to less than a dozen transactions per second [37] and they cannot scale by tuning their parameters [15].

More recent protocols [13, 38, 20, 1, 22, 25, 24, 32, 6] overcome the consensus bottleneck, but they are limited by the processing capacity of the nodes comprising the system. We provide background in §2.

Each node is required to store and process each block broadcasted on the network. As the number of transactions grows, and the longer the blockchain exists, it becomes increasingly difficult for nodes to keep up. Existing designs, where each node is operated on a single machine are not equipped to scale with the rising demand.

Several recent solutions [28, 26, 39, 2] propose to shard the blockchain, splitting it into multiple interleaved subchains. Each subchain is maintained by a distinct set of nodes, and so each node does not have to process all transactions in the system but only those in its subchain. This allows the system to process transactions at a higher rate while reducing the load on the individual node.

Those solutions target democratic environments, where there are many participants with similar resources. In such environments, increasing the number of participants allows to increase the number of subchains and thus system capacity. However, we show that when resource distribution is not democratic, as is the case in most popular cryptocurrencies, the security and performance of those systems deteriorate. We show how they could be vulnerable to attacks, and why they do not necessarily provide the benefits of lowering the burden on the single node. We discuss these solutions and related work in §3.

In this work, we present Ostraka, a novel architecture that allows for linear scaling of blockchain nodes capacity to meet network requirements. Ostraka enables scaling without forfeiting security assumptions. Rather than partitioning the system’s state into separately-managed clusters, in Ostraka we scale the nodes themselves, splitting each node across multiple Node-Shards (NSs), allowing vertical scaling of each node. The challenge is creating a partitioning scheme that achieves several goals. Data must be accessible in $O(1)$ while keeping the system secure and preventing an adversary from slowing down the validation process across the network. We describe the architecture in §4.

The implication is that nodes require more resources, so fewer entities will run such nodes (e.g hundreds instead of thousands currently in Bitcoin [14] and Ethereum [19]). We argue, however, that the same is true for subchain based proposals, where only an optimistic democratic setup would lead to better decentralization. In more realistic setups, with similar properties to those in active systems today, they would also imply a smaller number of full participants. Importantly, the decrease in node count does not mean that the system becomes closed: Anyone can still join the system and participate in the protocol. Meanwhile, as with current cryptocurrencies, end-users who send and receive payments can still use their mobile devices by querying a few nodes.

Ostraka is a node architecture and is thus independent of the underlying consensus protocol. We achieve improved throughput by applying a novel block validation algorithm, that allows even distribution of all aspects of block validations across multiple machines (§4). We demonstrate how we achieve these properties, without formfitting existing security guarantees.

We prove that the security guarantees of the protocol are the same as those of a theoretical high-performance unified (standard) node (§5). In contrast to existing solutions [29, 35], Ostraka does not regard transaction order within each block. This avoids expensive locking mechanisms requiring elaborate communication. We prove the correctness of this crucial enabler of distributed execution and show that the Ostraka distributed node is operationally indistinguishable from a unified-node, despite order changes and processing parallelization. Next, we show that an attacker can only perform a denial-of-service (DoS) attack against a small number of Ostraka nodes with high probability.

We evaluate Ostraka’s block propagation in a variety of configurations and network conditions. We show that an Ostraka node’s processing rate grows nearly optimally with the number of NSs. We provide theoretical analysis for the node performance as a function of inter- and intra-node bandwidth. We measure the performance of Ostraka in transaction processing with as many as 64 NSs, achieving a rate of nearly 400$k$ transactions per second. We demonstrate the effect of Ostraka on a full system through simulation of Bitcoin-NG [20], achieving an order of magnitude improvement. Evaluation details are in §6.

In summary, our main contributions are:

• •

Analysis of previous scaling solutions in non-democratic environments;

• •

Ostraka – a novel blockchain scaling architecture;

• •

Proof that Ostraka does not reduce security compared to a unified node architecture;

• •

Theoretical performance analysis showing linear scaling – up to network limits;

• •

Node implementation performance matching theory;

• •

A full system simulation.

2 Preliminaries: The UTXO Model

Ostraka utilizes the UTXO model [4, 30, 9] that was introduced in Bitcoin and is used in many systems. We describe here the elements relevant to this work.

Transaction

Each transaction comprises two lists – inputs and outputs. The outputs are the new records indicating ownership of amounts. Each transaction is uniquely identified by a hash of its contents,  TxHash. An input of a transaction references an output it spends by the TxHash of the transaction containing the output and its index in its output list. To spend an output, a spending condition must be satisfied, usually providing a cryptographic signature.

Node data structures

To operate a blockchain system, each of the nodes maintains three data structures: a set of transactions that were not yet placed in the blockchain – the mempool; the blockchain itself, a series of blocks containing transactions; and the current system state – the UTXO set.

Users create transactions and broadcast them, and the nodes store them in their mempools. Once a miner constructs a block from transactions in its mempool, she publishes the block to its peers. When the node receives a new block, it removes all outputs spent by transactions in the block from the UTXO set and adds the newly created outputs. The state of the UTXO set is thus the state of the blockchain following each block.

Security in Proof-of-Work Blockchains

In a PoW blockchain, block propagation time directly affects system security. Nakamoto consensus states that the most up-to-date state of the blockchain is the chain with the most work (the longest chain). If block propagation is slow, it takes time until other miners become aware of the new block. So two (or more) miners might generate blocks with the same predecessor. This split is called a fork.

To prevent forks, block propagation time must be much shorter than the interval between blocks. However, if nodes do not validate blocks before propagating them further, the system could be susceptible to DoS attacks, by flooding the system with invalid blocks. Reducing block processing time is therefore critical to blockchain security [17, 15, 21].

3 Sub-chain Analysis and Related Work

We overview current approaches to scaling blockchain performance, starting with novel security and performance analysis of proposals to split the system into subchains.

3.1 Concurrent subchains (Sharding)

With this approach, the system is divided into serveral subchains. Each subchain is maintained by a committee running a Byzantine Fault Tolerant (BFT) protocol, tolerant of an adversary of up to 1/3 of the network. Elastico [28] was one of the first proposals of this scheme. OmniLedger [26] uses subchains and builds on top of ByzCoin [25]. RapidChain [39] closely follows the ideas presented in Elastico and OmniLedger but proposes different solutions to committee election and cross-subchain transactions, not dependent on the user. Both OmniLedger and RapidChain use the UTXO model to distribute transactions across subchains. Chainspace [2] suggests a similar solution for a smart contracts system [37].

DoS vulnerability

These systems can be vulnerable to DoS attacks. Transactions are assigned to subchains according to their TxHash. An attacker can overwhelm a single subchain with many transactions much more cheaply than saturating the entire system. Other subchains are also throttled due to transactions that originate from multiple subchains.

We evaluate the effectiveness of this attack. Consider a system of $n$ subchains and transactions with a single output and $k$ inputs. The probability of a transaction to either have its output or one of its inputs in the subchain undergoing the attack is the complement of the probability that the output and all inputs not in the attacked subchain, $1-\left(\frac{n-1}{n}\right)^{k+1}$. Note that this is a lower bound: The number of transactions not able to execute is even higher when considering transaction dependant on pending transactions. Fig. 2 illustrates this result. It shows, for example, that even with 2-3 inputs, in a 2-subchain system about $90\%$ of transactions are affected, but even with 16 subchains about $20\%$ of the transactions are affected.

Performance

Sub-chain systems use various mechanisms for cross-subchain transactions. OmniLedger [26] uses 2-phase commit; it requires the client to obtain proofs of spending from each subchain where she wants input to be spent. RapidChain [39] relies on the leader of the output subchain to create new outputs, mirroring outputs spent on input subchains. In both approaches, since most transactions affect multiple subchains, the additional overhead is considerable. The architecture of Ostraka avoids this limitation.

Node load

Ostraka nodes distribute the data internally, such that every node maintains the entire state. This is seemingly a difference from subchain systems where each node only maintains a subset of the state. However, as we show below, this difference only holds in what we call democratic environments, where there are many miners (stakeholders for PoS) with similar mining power [26, 28, 39]. In non-democratic environments, where the number of miners is small, subchain systems also require nodes to store the entire state. Prominent systems relying either on PoW or PoS are typically non-democratic [16, 21], and recent findings [27] shows that indeed true decentralization is not possible in any Sybil protection mechanism used today. As of today, in Bitcoin, for example, about 85% of the mining power is controlled by 10 mining entities [10].

Recall that subchain systems rely on BFT protocols to maintain each of the subchains. The underlying BFT protocol assumes a bound on the number of Byzantine nodes, where a node is a unit of mining power. Even if the bound holds for the entire system (e.g., less than a third of the mining power belongs to malicious principals), in order to guarantee the bound, there should be sufficiently many nodes in each subchain [26]. With a 25% adversary, each subchain should have at least 600 nodes [26]. Studies [21] show that PoW systems are as decentralized as 20 nodes operating in a quorum, a far cry from the hundreds required for good security. Since the number of principals is not necessarily this large, participants instead split their mining power such that every mining power unit allows its owner one virtual node or share in the system. To prevent an attacker from targeting a specific subchain, each share is assigned to a subchain uniformly at random.

We estimate the minimal miner size for which a miner will have shares in most of the subchains. Denote by $S$ the number of nodes per subchain and by $n$ the number of subchains. Let $r$ be the target ratio of subchains that should be occupied in expectation and denote the miner size to achieve the target by $\alpha$. The number of shares belonging to the miner is $\alpha nS$. The expected number of subchains occupied by the miner is therefore $n\left(1-\left(\frac{n-1}{n}\right)^{\alpha nS}\right)$. This expression should equal the target number of subchains, $rn$, and solving for $\alpha$ we get $\alpha=\frac{\log(1-r)}{Sn\log(1-1/n)}$.

To obtain concrete numbers, we take $S$ to be either 300 or 600 as required by Kogias et al. [26] and vary the number of subchains, $n$, from 1 to 32. We target an expected $90\%$ of the subchains and obtain the results shown in Fig. 2. Indeed, even with only 300 nodes per subchain and 32 subchains, a miner would participate in $90\%$ of the subchains in expectation if its size is larger than $0.8\%$

The implication is that in non-democratic scenarios, which are common in active blockchains, the miners must participate in all subchains, requiring them to process the entire chain, therefore not achieving the goal of reducing stress on the single node.

3.2 Node scaling proposals

Dickerson et al. [18], implement concurrent execution in smart contracts in a virtual machine model as used in Ethereum [37]. They add a concurrent schedule to each block, enabling consistent parallel execution across different nodes. However, the nature of the VM model limits the level of their measured speed up to 2x. Read-after-Write dependencies of instructions in the VM model enforce a sequential execution to avoid an inconsistent state. Ostraka does not suffer from these limitations; we observed an improvement of two orders of magnitude, limited only by the network and the capacity of our experimental testbed.

4 Architecture

We proceed to describe the structure and operation of Ostraka. An Ostraka node operates similarly to a unified node. The key difference is that in Ostraka the mempool, UTXO set, and blockchain storage are distributed among several machines, called node-shards (NSs).

We describe the system components (§4.1), explain the techniques used to make it efficient and secure (§4.2), and overview system operation (§4.3).

4.1 System Components

In Ostraka a coordinator keeps track of the blockchain and orchestrates communication within the node (intra-node) and with other nodes (inter-node). We refer to the NSs of the same node as sibling NSs. The coordinator can operate as a separate machine or as a process on one of the NSs. In Ostraka work distribution is local on each node and does not mandate the same number of NSs per node. Transactions are distributed among NSs as atomic units. Transaction outputs are stored in the UTXO-shard of the NS they are assigned to. Note we assume complete trust among the NSs and the coordinator within a node.

Coordinator

The coordinator orchestrates the node operation. When the node is first initialized, all NSs connect to the coordinator. The coordinator coordinates communication between sibling NSs and with other nodes.

The coordinator tracks the state of the blockchain. It stores the header of each block and determines the main chain (e.g., with most PoW in Bitcoin and Ethereum). It instructs the NSs of which blocks to request, process, and validate. In the case of a reorg (rollback of state, which is possible in some blockchains), the coordinator commands the NSs to roll back their state to the block of the divergence and start following the new chain.

Node-Shards

Each NS is assigned a local ShardID. The ShardID determines which transactions the NS is responsible for. Once all NSs are connected, the coordinator sends each NS the ShardID, IP address, and port of all sibling NSs. Next, NSs connect to form a clique. Via the ShardID, each transaction is mapped to the NS responsible for storing it and its outputs.

Each NS stores transactions in a block-shard – a portion of a block that contains only transactions assigned to its ShardID. It similarly stores UTXOs in a UTXO-shard, and pending transactions in its mempool shard.

4.2 System Design

We proceed to describe the details of the components’ interaction. The design sets to achieve several goals that seem contradictory. First, transaction lookup should be an $O(1)$ operation so block validation is quick. Transaction to NS assignment should allow for efficient communication between NSs. Nevertheless, an adversary should not be able to overload any single NS. Ostraka achieves all these goals as follows.

Transaction inputs reference previous transaction outputs by TxHash and the index of the output in the transaction. Each TxHash is unique to a transaction (we use SHA-256). Therefore, TxHash is the natural index for transactions.

Any-order execution

In unified blockchain clients, storage and processing are all performed on a single machine (e.g., [29, 37]), transactions in a block are ordered as a list. A transaction can only spend outputs created in a previous block or earlier in the same block’s list. We call this common approach, the topological-order algorithm. Using a topological order in Ostraka would have implied unnecessary dependencies, requiring costly coordination among NSs. Instead, Ostraka treats each block as an unordered set of transactions. Dependencies of a transaction are satisfied if the referenced output is in an earlier block or anywhere in the current block. We justify the correctness of unordered block validation in Section §5.

Adding salt

When considering how to distribute transactions equally among NSs, one option to simply take the TxHash of each transaction, and map it to a shard using a deterministic function (e.g. taking the MSBs of the hash). This method relies on the natural distribution of the SHA-256 value of each transaction. However, this would allow an attacker to cheaply form transactions that would all be assigned to the same NS on all nodes. The processing speed of all nodes would then be $1/\ell$ (for $\ell$ NSs), achieving DoS. We further analyzed DoS susceptibility in Section 5.2.

To prevent such attacks, each node generates a random salt upon initialization. We use the salt and apply it to each transactions with an additional hash as follows: $\texttt{NewTxHash}=\textsc{SHA-256}(\texttt{TxHash}||{salt})$. By applying a deterministic function to the NewTxHash, we distinctly map a transaction to a NS, Outputs in the UTXO set are stored with the original TxHash, while the NewTxHash is used internally to distribute transaction between NSs.

The coordinator publishes the chosen salt to its NSs and remote nodes. Thus, peers know the NewTxHash for the specific salt of the peer, and only sends transactions relevant to the requesting NS’s ShardID. This way network bandwidth is distributed among the NSs and each NS receives only transactions it requires. Direct NS to NS communication between nodes avoids bottlenecks.

Note

Routing all transactions through the coordinator would allow us to keep the salts secret, making DoS attacks harder. But as expected, our experiments show that with this design the coordinator becomes a bottleneck.

4.3 Operation

We proceed to describe networking and Ostraka’s distributed block validation.

Establishing connections

Nodes connect to each other according to the system’s topology rules (e.g., several random peers in Bitcoin). When the rules indicate that node $A$ should connect with node $B$, the two coordinators connect, and all NSs of node $A$ connect to all NSs of node $B$. An example is presented in Fig 3

Block validation

When a node receives a new block, it sends an Inventory message to its peers, informing them that a new block has been produced. The receiving coordinator informs its NSs of the new block hash and the peer who has it. Each NS requests the relevant transactions from all the NSs of the peer. It sends the peer NSs its local ShardID and the block hash it is interested in. Each NS of the sending node iterates over the transactions associated with the block hash and calculates the NewTxHash using the salt. The sending NS sends the requester only relevant transactions.

Block validation comprises block header validation, where most operations are performed in $\mathcal{O}(1)$, and transaction validation, where operations are $\mathcal{O}(n)$ ($n$ is the number of transactions in the block). Block header validation is performed by the coordinator. It checks for duplicates, PoW, correct reference hash, etc.

The exception is Merkle root validation. The coordinator performs Merkle root validation which is linear in $n$. First, NSs calculate the hashes of their transactions, the leaves of the Merkle tree, and send them to the coordinator. On a node with $\ell$ NSs and a block with $n$ transactions, each NS is performing $n/\ell$ hash operations. The coordinator lexicographically sorts the transactions by hash and then calculates the remaining $n$ hashes to build the root. This process could in principle be delegated to NSs by replacing the Merkle tree with an accumulator [8, 7], but their group operations result in performance that is significantly worse than the Merkle tree calculation.

NSs perform their part of block validation concurrently with the coordinator. The coordinator awaits approval from all NSs before adding the current block to the chain. If at any point one of the NSs comes across an illegal transaction, it sends a BadBlock message to the coordinator, who then discards the block.

Fig. 4 summarizes the steps of block propagation and validation. To validate a transaction, a NS must first obtain all the required information. A NS only has its share of UTXOs in the UTXO set. It constructs a list of missing outputs to request from each sibling NS by using the TxHash, the salt, and the ShardID. Requests to each NS are sent in a batch. Upon request, each NS sends the transaction output to the requester.

Once an output has been requested, it is marked as spent. Multiple requests for the same output indicate a double-spend attempt, and will thus trigger a BadBlock message to the coordinator. Once it receives all transaction outputs, the NS proceeds to validate the scripts. These checks include:

• •

Equal sum: For each transaction, the aggregate amount of inputs is at least as high as the outputs.

• •

No double spends: Each input references an output that was produced but not spent.

• •

Authorization: Each input complies with the spending condition of the output it consumes.

Other operations

Other inter-node operations are easily implemented in Ostraka. Peers can request transaction directly from NSs, and block headers from the coordinator.

One special case is that of a reorg. A reorg happens when a node needs to switch the chain it follows. The coordinator and NSs must roll back to a state of a previous block, then update it to the longest chain.

A reorg is performed as follows: First, the coordinator sends the hashes of blocks to roll back, as well as the hashes of the new blocks. Each NS fetches the spent outputs records of the rolled-back block from the database. Each NS adds the spent outputs of each transaction in the block to the UTXO-shard. Next, it removes all outputs created by transactions in the blocks-shard from the UTXO-shard. No communication is required among the NSs during the rollback, and it can thus be quickly resolved.

5 Security

Ostraka can be used in conjunction with many consensus protocols (e.g., [38, 20, 25, 24, 32]). We thus need to prove that a system using Ostraka is at least as secure as a system using a unified node.

We start by showing that Ostraka behaves as a unified node (§5.1). Beyond functionality, blockchain protocols rely on propagation time to maintain security. Therefore we also show that Ostraka’s design does not enable attacks on node processing time. We show how individual-node shuffling prevents an adversary from affecting macro system performance (§5.2).

5.1 Functional Indistinguishability

Blockchain systems typically [29, 37, 25, 20] form blocks that are ordered such that each transaction follows transactions it depends on. We call this a topological-order System.

In contrast, Ostraka is agnostic the internal order. We call such a system an unordered system. Of course, topologically ordered blocks may be invalid in a system that expects lexicographically ordered blocks, and vice versa. The Ostraka algorithm would except a block created by Bitcoin miners, but not every out-of-order block, would be accepted in Bitcoin without changes to the validation rules. We thus need to show that any set of transactions can be processed as a legal block in a topological system if and only if they can be processed as a legal block in the unordered system. We leave it for future work to explore if Ostraka can be integrated with Bitcoin as it exists today. As the resulting Merkle root depends on transaction ordering, this might not be possible without a hard fork.

We show that an unordered system is functionally indistinguishable from a Topological-order system.

From topological to unordered systems

Most stages of block validation (cf. §4.3) like signature and PoW verification are agnostic to transaction order. Two exceptions are the Merkle tree calculation and transaction dependency checks.

Merkle tree calculation depends on the transaction order, but the choice of the order does not matter as long as all nodes calculate it consistently. For this purpose, Ostraka orders transactions lexicographically, according to their TxHash.

It remains to show that any set of transactions can be processed as a legal block in a topological system if and only if they can be processed as a legal block in the unordered system.

To formally state this claim we first specify the two block-validation algorithms. Both take a UTXO set and a set of transactions and return either the updated UTXO set or $\bot$ if the transactions cannot form a legal block.

The topological-order system algorithm denoted $\mathcal{A}_{T}$, is shown in Algorithm 1. The algorithm receives a UTXO_set and the set of transactions $T$. The algorithm first sorts $T$ topologically and then processes the transactions in order. It is important to note that the set $T$ can always be ordered topologically: cycles between transactions cannot form since the references are by transaction hash. Assuming SHA-256 can be modeled as a random oracle, Harris [23] shows that for a space of $2^{\kappa}$, the expected number of steps to find a cycle is $\frac{1}{4}\sqrt{2\pi 2^{\kappa}}$, therefore an adversary cannot find such a cycle except with negligible probability in the hash output size. Therefore the sorting algorithm is well defined.

The algorithm returns $\bot$ if at some point it encounters a transaction whose input is in not in the UTXO_set. For each transaction, the operation Valitates TX (line 1), covers all additional transaction validation steps after the referenced outputs were found, e.g., script validation, correct sum etc. If Validate TX fails, it returns $\bot$.

The unordered system algorithm denoted $\mathcal{A}_{U}$, is shown in Algorithm 2. Unlike Algorithm $\mathcal{A}_{T}$, in $\mathcal{A}_{U}$ transactions are not ordered. All outputs are first added to the UTXO_set, then the algorithm processes the transactions sequentially in an arbitrary order. Other than that, the operation is similar to $\mathcal{A}_{T}$.

We show that Algorithms $\mathcal{A}_{T}$ and $\mathcal{A}_{U}$ are functionally indistinguishable.

Lemma 1**.**

Given the same UTXO_set and set of transactions $T$, $\mathcal{A}_{T}$ and $\mathcal{A}_{U}$ return the same value.

We prove the lemma by showing that $\mathcal{A}_{T}$ returns $\bot$ if and only if $\mathcal{A}_{U}$ returns $\bot$:

Proof.

We prove in both directions.

$\mathcal{A}_{T}\to\mathcal{A}_{U}$

We first show that if $\mathcal{A}_{T}$ rejects an input tuple $(T,\texttt{UTXO\_set})$ then $\mathcal{A}_{U}$ also rejects this input.

If $\mathcal{A}_{T}$ returns $\bot$, there exists some transaction $\textit{TX}_{i}\in T$, at index $i$, and an input in $\textit{TX}_{i}$, referencing an UTXO $\textit{Ref\_Out}\not\in\texttt{UTXO\_set}$, produced by an output TX_out. This can be a result of three cases, and we show that $\mathcal{A}_{U}$ rejects as well.

The UTXO is not in the UTXO_set and is not produced by any transaction preceding $\textit{TX}_{i}$: There can be no transaction succeeding $\textit{TX}_{i}$ which produces TX_out as it contradicts the topological order, thus, there is not any transaction producing output TX_out in $T$. Upon processing $\textit{TX}_{i}$, $\mathcal{A}_{U}$ return $\bot$ (Algorithm 2 line 2). 2. 2.

The UTXO is not in the UTXO_set and is produced by some transaction $\textit{TX}_{k}$, preceding $\textit{TX}_{i}$ but consumed by some transaction $\textit{TX}_{j}$, succeeding $\textit{TX}_{k}$ but preceding $\textit{TX}_{i}$: In $\mathcal{A}_{U}$, TX_out is added to the UTXO_set in line 2, $\mathcal{A}_{U}$ processes either $\textit{TX}_{j}$ or $\textit{TX}_{i}$ first. Without loss of generality, $\textit{TX}_{i}$ is processed first. Upon processing $\textit{TX}_{j}$, $\mathcal{A}_{U}$ would not find UTXO Ref_Out in the UTXO_set and return $\bot$ (Algorithm 2 line 2). 3. 3.

The UTXO is in the UTXO_set and it is consumed by some transaction $\textit{TX}_{j}$, preceding $\textit{TX}_{i}$: Similarly to the previous case, except now the UTXO is in the UTXO_set.

$\mathcal{A}_{U}\to\mathcal{A}_{T}$

If $\mathcal{A}_{U}$ returns $\bot$, there exists some transaction $\textit{TX}_{i}\in T$, and an input Ref_Out in $\textit{TX}_{i}$, referencing an UTXO $\textit{Ref\_Out}\not\in\texttt{UTXO\_set}$ produced by an output TX_out. This can also be a result of three cases, and we show that $\mathcal{A}_{T}$ rejects as well.

The UTXO is not in the UTXO_set and is not produced by any transaction in $T$: In $\mathcal{A}_{T}$, the UTXO is not produced by any transaction in $T$ and specifically, by any transaction preceding $\textit{TX}_{i}$ in the topological order of $T$ in $\mathcal{A}_{T}$. Thus $\mathcal{A}_{T}$ return $\bot$. 2. 2.

The UTXO is not in the UTXO_set and is produced by some transaction $\textit{TX}_{k}$, but is also consumed by some transaction $\textit{TX}_{j}$ : In the topological order of $T$, both $\textit{TX}_{j}$ and $\textit{TX}_{i}$ must succeed $\textit{TX}_{k}$. Transaction $\textit{TX}_{k}$ is processed first and adds Ref_Out to the UTXO_set. Without loss of generality, $\textit{TX}_{i}$ is processed first and removes Ref_Out from the UTXO_set. Upon processing $\textit{TX}_{j}$, output Ref_Out is not in the UTXO_set and system $T$ return $\bot$ (Algorithm 1 line 1). 3. 3.

The output is in the UTXO_set and it is also consumed by some transaction $\textit{TX}_{j}$: Similar to the second case, but Ref_Out is in the UTXO_set.

Therefore, given the same UTXO_set, and transactions $T$, $\mathcal{A}_{T}$ return $\bot$ if and only if $\mathcal{A}_{U}$ returns $\bot$. Both algorithms add the same set of outputs to the UTXO_set, and remove the same set of utxos. No output can be added to the UTXO_set twice, and any attempt to remove an output twice results in $\bot$. Thus the new UTXO_set returned by each algorithm is identical. ∎

From unified node to distributed node

We now show that Ostraka is functionally indistinguishable from $\mathcal{A}_{U}$. The distributed algorithm of Ostraka, denoted $\mathcal{A}_{O}$, is shown in Algorithms 3–4. A Coordinator algorithm receives the UTXO_set and transaction set $T$ as input. The Coordinator algorithm first distributes the UTXO_set and transactions among $\ell$ NSs. NSs begin validation by adding all produced outputs in their set of transactions $T_{i}$, to their local UTXO_shard. They proceed by requesting missing output information from the relevant siblings. Once they receive the information, each NS proceeds to validate its set of transactions. If some validation stage fails the NS sends $\bot$, otherwise the NS sends its updated UTXO_shard. If all NSs returned updated sets, the coordinator returns their union, otherwise $\bot$. The pseudo-code makes a slight simplification of Ostraka that nonetheless represents its logical behavior.

We show that an unordered-system algorithm and Ostraka are functionally indistinguishable.

Lemma 2**.**

Given the same UTXO_set and set of transactions $T$, $\mathcal{A}_{U}$ and $\mathcal{A}_{O}$ return the same value.

As before, we prove the lemma by showing that $\mathcal{A}_{U}$ returns $\bot$ if and only if $\mathcal{A}_{O}$ returns $\bot$:

Proof.

We prove in both directions.

$\mathcal{A}_{U}\to\mathcal{A}_{O}$

We first show that if $\mathcal{A}_{U}$ rejects an input $(T,\texttt{UTXO\_set})$ then $\mathcal{A}_{O}$ also rejects this input.

If $\mathcal{A}_{U}$ returns $\bot$, there exists some transaction $\textit{TX}_{i}\in T$, and an input in $\textit{TX}_{i}$, referencing an UTXO $\textit{Ref\_Out}\not\in\texttt{UTXO\_set}$, produced by an output TX_out.

The UTXO is not in the UTXO_set and is not produced by any transaction in $T$: The UTXO is not produced by any transaction in $T$, thus no NS receives a transaction producing Ref_Out in  $\mathcal{A}_{O}$. WLOG, some NS $S_{i}$ receives $\textit{TX}_{i}$. If Ref_Out references an output assigned to a NS $S_{j}$, when receiving request for Ref_Out, $S_{j}$ does not have Ref_Out in its UTXO_shard and send $\bot$ (Alg. 4 line 4). If Ref_Out references a transaction assigned to $S_{i}$, when processing transactions, Ref_Out is not in UTXO_shard of $S_{i}$, and $S_{i}$ sends $\bot$ (line 4). 2. 2.

The UTXO is not in the UTXO_set and is produced by some transaction $\textit{TX}_{k}$, but is also consumed by some transaction $\textit{TX}_{j}$: WLOG, TX_out is produced by a transactions assigned to some NS $S_{k}$, and is added to its UTXO_shard (Algorithm 4 line 4). If both consuming transactions $\textit{TX}_{j}$ and $\textit{TX}_{i}$, are assigned to sibling NSs, then upon receiving all requests from sibling NSs, $S_{k}$ processes one of the requests first and remove Ref_Out from it UTXO_shard. Upon processing the second request, Ref_Out is not in UTXO_shard, and $S_{k}$ sends $\bot$ (Algorithm 4 line 4). If $\textit{TX}_{j}$ is assigned to a sibling NS, and $\textit{TX}_{i}$ is assigned to $S_{k}$, Ref_Out is removed from UTXO_shard after all outputs are requested. Upon processing $\textit{TX}_{i}$, $S_{k}$ does not have Ref_Out in UTXO_shard, and send $\bot$ (Algorithm 4 line 4). If both $\textit{TX}_{i}$ and $\textit{TX}_{j}$ are assigned to $S_{k}$, one of them is be processed first and removes Ref_Out from the UTXO_shard, and upon processing the second one, Ref_Out is not be in UTXO_shard and $S_{k}$ sends $\bot$ (Algorithm 4 line 4). 3. 3.

The UTXO is in the UTXO_set and it is also consumed by some transaction $\textit{TX}_{j}$: This is similar to the second case, but TX_out is in the UTXO_shard of $S_{k}$.

$\mathcal{A}_{O}\to\mathcal{A}_{U}$

If $\mathcal{A}_{O}$ returns $\bot$ and rejects, then some NS $S_{i}$ sends $\bot$ to the Coordinator. This can be a result of several cases, and we show that $\mathcal{A}_{U}$ rejects as well.

First, notice that the UTXO_set and transactions $T$ are distributed deterministically. There can be no two NSs producing the same TX_out, and similarly, Ref_Out cannot reside in two UTXO-shards.

When processing requests for outputs, there is some UTXO $\textit{Ref\_Out}\not\in\texttt{UTXO\_shard}$

1. (a)

The UTXO is not in the UTXO_shard, is not produced by any transaction in $T_{i}$, and is requested by some sibling NS $S_{j}$. Then there exists some transaction $\textit{TX}_{j}$ $\in T$ consuming UTXO. As UTXO_set is distributed deterministically, there can be no other UTXO_shard other than $\texttt{UTXO\_set}_{i}$ where the UTXO might reside. Therefore, it is not in the UTXO_set of $\mathcal{A}_{U}$, and upon processing $\textit{TX}_{j}$, $\mathcal{A}_{U}$ returns $\bot$ (Algorithm 2 line 2). 2. (b)

The UTXO is not in the UTXO_shard, is produced by some transactions $\textit{TX}_{k}$, and is requested by some two sibling NS $S_{j}$, $S_{l}$. Then there exist some two transaction $\textit{TX}_{j}$ and $\textit{TX}_{j}$ $\in T$, consuming the same UTXO. WLOG, in $\mathcal{A}_{U}$, $\textit{TX}_{j}$ is processed first and consumes the UTXO. Upon processing $\textit{TX}_{l}$, $\mathcal{A}_{U}$ returns $\bot$ (Algorithm 2 line 2). 2. 2.

When processing transactions, there exists some transaction $\textit{TX}_{i}\in T_{i}$, and an input in $\textit{TX}_{i}$, referencing an UTXO $\textit{Ref\_Out}\not\in\texttt{UTXO\_shard}$.

1. (a)

The UTXO is in the UTXO_shard and is requested by some sibling NS $S_{j}$: There is some other transaction $\textit{TX}_{j}$ $\in T$ consuming the UTXO. WLOG, in $\mathcal{A}_{U}$ $\textit{TX}_{j}$ is processed first, and upon processing $\textit{TX}_{i}$, $\mathcal{A}_{U}$ returns $\bot$ (Algorithm 2 line 2). 2. (b)

The UTXO is not in the UTXO_shard, is produced by some transaction $\textit{TX}_{k}$, and is requested by some sibling NS $S_{j}$. The case is similar to the previous after all outputs are added to the UTXO_set in Algorithm $\mathcal{A}_{U}$ (Algorithm 2 line 2). 3. (c)

The cases enumerated for $\mathcal{A}_{U}$ are the same if all transactions are assigned to a single NS, and are not requested by any sibling.

Therefore, given the same UTXO_set, and transactions $T$, $\mathcal{A}_{U}$ return $\bot$ if and only if $\mathcal{A}_{O}$ returns $\bot$. ∎

We can now show that a classical topological-order system with a unified node is functionally indistinguishable from an Ostraka node.

Theorem 1**.**

Given the same UTXO_set and set of transactions $T$, a topologically ordered system and Ostraka, are functionally indistinguishable.

The proof follows directly from Lemmas 1 and 2.

5.2 Processing Time

We have shown the systems are functionally indistinguishable; it remains to show that an attacker cannot affect block processing time in Ostraka more than it could in a unified node. Limiting processing time is important to prevent an attacker from strangling the system, creating a major slowdown and loss of throughput.

Targeting a single NS

In a unified node, block processing time is (roughly) the sum of processing all transactions. In Ostraka it is the time it takes the slowest NS to process its transactions.

If transaction distribution is identical for all nodes, then an attacker can cheaply generate transactions that are all placed in the same NS. For example, an attacker can create a block where all transactions start with multiple zeros in the MSB. In general, to create a transaction starting with $\ell$ zeros, the attacker needs $2^{\ell}$ attempts in expectation. For a block of $L$ transactions, a total of $L\cdot 2^{\ell}$ attempts are required in expectation. Note that this analysis applies to any index function.

We denote by Df the distribution function applied to TxHash to determine the NS responsible for it. For $\ell$ NSs, the Df results in a number from $1$ to $\ell$.

In Ostraka, we use a salt to redistributed transactions. Each node is aware of the salt values of its neighbors. As connections are performed randomly, it is unlikely a node will obtain the values of all nodes in the system. Yet, even if one obtains all salts, attack difficulty rises exponentially with the number of targeted nodes.

Assume the attacker knows the salt values of $k$ peers, $S_{1}..S_{k}$. We define the distribution function Df for a value $x$ and a salt value $S_{i}$ to be ${\textit{Df}(x,S_{i})~{}=~{}\textsc{SHA-256}(x||S_{i})}$. The first transaction the attacker creates will result in some value $v_{i}$ for each of the $k$ peers. The attacker now needs every subsequent transaction to result in the same value $v_{i}$ for each peer. The attacker thus needs to find a TxHash such that: ${\forall 1\leq i\leq k:\textit{Df}(\texttt{TxHash},S_{i})~{}=~{}v_{i}}$. The probability of finding such value, if the attacker tries to target one of $\ell$ NSs is $(1/\ell)^{k}$. To cause an $\ell$ times slowdown for $k$ peers, requires ${\ell}^{k}$ attempts per transaction in expectation.

It quickly becomes infeasible to affect even a small percentage of the system. For example, creating one transaction affecting the validation time of 100 nodes requires the hashing rate of Bitcoin for 800,000 years [11].

Creating a single large transaction

Creating an abnormally large single transaction occupying an entire block can also cause a significant slowdown. In Ostraka, each transaction is processed by a single NS. A huge transaction that occupies an entire block, would annul the benefits of the distributed architecture. Adding salt does not help mitigate this attack vector. In Ostraka, we mitigate this attack by adding a transaction size limit. The limit should be proportional to the block size, to allow even distribution of transactions across multiple NSs. In general, a user can make any payment with only two types of transactions. A join transaction of two outputs into one, or a fork transaction of a single output into two.

6 Evaluation

The throughput of a blockchain depends on $t_{\textit{hop}}$: the time it takes for a single node to receive and process the block. $t_{\textit{hop}}$ depends on both block transmission time $Btt$ and block processing time $Bpt$. Each node validates the block before it propagates it further, to prevent DoS attacks on the system [15, 17].

We present the measurements and theoretical analysis of $t_{\textit{hop}}$. We begin by describing our evaluation methodology (§6.1), and proceed to measure how the number of NSs and the node bandwidth affects the $t_{\textit{hop}}$ (§6.2). Similarly, we analyze and present the effect of intra-node bandwidth (§6.3). We conclude with a network simulation (§6.4).

6.1 Methodology

We have implemented the necessary elements for performance evaluation based on the existing btcd Bitcoin client [12]. We implement the intra-node and inter-node networking, as well as our distributed block validation protocol. In total, we have added  10K LoC in Go and an additional 2K LoC of Python scripts for testing and simulation. Our implementation is available as open source [3].

We measure block sending time between two Ostraka nodes and block processing time on the receiving node. We used a c4.large EC2 instance (2 vCPUs 4GB RAM) for each NS and a c4.xlarge instance (4 vCPUs 8GB RAM) for the coordinator of each node. A block comprises transactions with two inputs and two outputs, i.e., about 250 Bytes, which is the median Bitcoin transaction size. Each single EC2 NS can process approximately $1700\ \textit{Tx}/\textit{sec}$. Each block undergoes full validation, i.e. we do not assume some of the transactions have been previously known to the receiving node, and have already been validated. Transaction dependency within the block does not affect processing time, as transaction validation is performed in distributed, achievable by our block validation algorithm. Each figure with error bars presents the minimum, maximum, and average measurements. Theoretical analysis is presented where applicable.

6.2 Block time and node capacity

Total block time $t_{\textit{hop}}$ depends on both block transmission time $Btt$ and block processing time $Bpt$. We begin by isolating and evaluating the $Bpt$. We measure $Bpt$ of various configurations, without restriction on intra-node or inter-node bandwidth.

The number of transactions in a block is the block size, denoted BSize, divided by the mean transaction size, denoted TxSize. $\textit{NS}_{\textit{tps}}$ is the number of transactions per second a NS can process. $Bpt$ is thus the number of transactions in a block divided by all NSs combined processing rate : $Bpt={\frac{\textit{BSize}}{\textit{TxSize}}\cdot\frac{1}{\textit{NS}_{\textit{tps}}\cdot\textit{\#{{NS}}s}}}$.

We measure the $Bpt$ with an increasing number of transactions. We repeat this experiment varying the number of NSs from 1 to 32. We estimate the number of transactions each configuration processes in ten seconds from a linear regression. We find the maximum block size a node is capable of processing within the limit. Ten seconds was chosen arbitrarily to be an upper limit on the processing time of each node. The two Unlimited plots of Figure 7 show our measurements. The theoretical unlimited bandwidth line represents the best achievable results, multiplying the processing capacity of a single NS by a factor. The error bars represent our actual measurements. We observe that Ostraka achieves near-optimal improvement as we increase the number of NSs.

Next, we add the effect of block transmission time ($Btt$) on $t_{\textit{hop}}$. $Btt$ is determined by network latency and bandwidth. The bandwidth, denoted BW, is the total network bandwidth available to all the NSs combined. We focus on network bandwidth as it is more significant than the latency for large blocks [15]. We define a node’s capacity as the number of total transactions per second a node can receive and processes. The capacity is thus the total number of transactions in a block $\frac{\textit{BSize}}{\textit{TxSize}}$, divided by $t_{\textit{hop}}$.

[TABLE]

By adding more NSs we expect $t_{\textit{hop}}$ to approach $Btt$, as $Bpt$ approaches 0. We measure capacity when both the network bandwidth and the number of NSs vary. We measure with configurations of 1,2,4,8,16 and $32$ NSs, as well as total bandwidth of $40$ and $160\textit{Mbit/s}$ for the combined node. As before, we estimate capacity by evaluating the linear regression at ten seconds.

Fig 7 shows capacity measurements as well as theoretical analysis. Error bars represent our measurements for each configuration of bandwidth and NSs. The dashed lines represent our theoretical analysis as shown in equation (1). Our measurements match our theoretical evaluation: as we increase the number of NSs, the capacity of the node increases, yet limited by the available bandwidth.

6.3 Intra-node network

Part of the Ostraka protocol is requesting and sending outputs between NSs. When intra-node bandwidth is high, time to send requests and later to send information to requesting NSs is not significant. Ostraka can be implemented to run on a single system, where NSs communicate over Unix sockets, with much higher bandwidth. However, we wanted to also examine the case of several distant NSs with limited bandwidth between them. Limiting the intra-node bandwidth would affect the $Bpt$.

For $n$ transactions per block and $\ell$ NSs, each NS has in expectation $\frac{n}{\ell}\frac{\ell-1}{\ell}$ outputs it needs to request from its siblings. This is also the number of outputs it is required to send.

It is important to note that although the number of siblings each NS must contact increases with the number of NSs, the overall number of outputs requested/sent per NS decreases. Thus, intra-node communication benefits and decreases linearly as we add additional NSs.

We denote by ${\textit{BW}_{\textit{NS}}}$ the intra-node bandwidth and by $\textit{TX}_{out}$, the expected size required for sending and receiving outputs information per transaction. $Bpt$, when accounting for intra-node communication is thus:

[TABLE]

We measure $Bpt$ while restraining intra-node bandwidth to $5\textit{Mbit/s},1\textit{Mbit/s},256\textit{Kbit/s}$ per NS. Similarly to previous experiments, we estimate processing capacity by measuring $Bpt$ for various block sizes. Since processing time is higher, we look at 30 seconds for our linear regression.

For $5\textit{Mbit/s}$, despite the limited bandwidth, the benefits of distributing transaction validation and storage outweigh the limits of cross-NS communication.

For $256\textit{Kbit/s}$ per NS, we observe a significant dip in performance when moving from a single NS to two NSs. This is the result of sending information across NSs becoming dominant. Fig. 7 shows these effects. Our theoretical evaluation falls within the margin of error. When intra-node bandwidth is extremely limited, it is more beneficial to operate a single NS.

6.4 Full System Simulation

Finally, to assess the benefits of increasing block validation rate on a blockchain system, we take as an example the Bitcoin-NG [20] protocol. We simulate 1,000 nodes implemented with Ostraka instead of unified nodes as in the original paper. Each node is connected to 8 other nodes at random, similar to Bitcoin. We target 10 minutes between leader election blocks and 10 seconds between microblocks, as Bitcoin-NG. We produce an event-driven simulation, with identical node configurations, a network topology similar to Bitcoin, and fixed bandwidth per node. Fig. 7 shows the effect of adding additional NSs to each node. Network topology and properties strongly affect the protocol’s performance, but the trend is clear. Ostraka allows the system to achieve an order of magnitude improvement in throughput.

Table I summarizes the comparison of Ostraka with Bitcoin-NG, OmniLedger, and RapidChain (cf. §3). The table compares results as reported by the two papers. We compare the adversarial percentage of the network the system is resilient against. In the case of Ostraka, we consider the Bitcoin-NG PoW setup we used in our simulation. Using the Bitcoin-NG consensus algorithm, the system is resilient against a 49% adversary. OmniLedger and RapidChain require a high number of nodes and participants to reach high throughput and good security guarantees. Ostraka, on the other hand, does not require a lower bound on the number of nodes. Deployed in democratic environments, where voting power is distributed equally across the network, Omniledger and RapidChain outperform Ostraka by sharding the bandwidth and the computational resources. However, in systems whose voting power is not uniformly distributed (e.g., Bitcoin, Ethereum), it is possible to apply the Ostraka design to reach high throughput.

7 Conclusion

We present Ostraka, an architecture for scalable blockchain nodes. Ostraka overcomes the security and performance limitations of previous solutions and achieves linear scaling in the number of node-shards. Our experiments reached a rate of $400k$ tx/sec at the capacity of our experimental testbed. Together with a high-performance consensus protocol, Ostraka can saturate network capacity.

Acknowledgment

This research was supported by the Israel Science Foundation (grant No. 1641/18), the Technion Hiroshi Fujiwara cyber-security research center, and the Israel cyber bureau.