Can 100 Machines Agree?

TL;DR

This study empirically evaluates the performance of various agreement protocols on up to 100 machines, revealing that throughput decay initially sharpens then dampens, with chain- and ring-based protocols outperforming others significantly.

Contribution

Provides the first extensive empirical analysis of agreement protocols at large scale, demonstrating their graceful performance and identifying advantages of chain- and ring-based methods.

Findings

Agreement protocols perform well on 100 machines with dampened throughput decay.

Chain- and ring-based protocols achieve 4K to 11K requests/sec, outperforming traditional protocols.

Performance decay slows down beyond a few tens of replicas, indicating scalability benefits.

Abstract

Agreement protocols have been typically deployed at small scale, e.g., using three to five machines. This is because these protocols seem to suffer from a sharp performance decay. More specifically, as the size of a deployment---i.e., degree of replication---increases, the protocol performance greatly decreases. There is not much experimental evidence for this decay in practice, however, notably for larger system sizes, e.g., beyond a handful of machines. In this paper we execute agreement protocols on up to 100 machines and observe on their performance decay. We consider well-known agreement protocols part of mature systems, such as Apache ZooKeeper, etcd, and BFT-Smart, as well as a chain and a novel ring-based agreement protocol which we implement ourselves. We provide empirical evidence that current agreement protocols execute gracefully on 100 machines. We observe that throughput decay is initially sharp (consistent with previous observations); but intriguingly---as each system grows beyond a few tens of replicas---the decay dampens. For chain- and ring-based replication, this decay is slower than for the other systems. The positive takeaway from our evaluation is that mature agreement protocol implementations can sustain out-of-the-box 300 to 500 requests per second when executing on 100 replicas on a wide-area public cloud platform. Chain- and ring-based replication can reach between 4K and 11K (up to 20x improvements) depending on the fault assumptions.