Spindle: Techniques for Optimizing Atomic Multicast on RDMA

TL;DR

Spindle introduces optimization techniques for atomic multicast over RDMA, significantly improving throughput and latency in applications like Derecho, and demonstrating broad applicability for RDMA-based systems.

Contribution

The paper presents novel batching, null-message, and synchronization techniques that enhance RDMA atomic multicast performance, with demonstrated improvements in throughput and latency.

Findings

Multicast bandwidth increased from 1GB/s to 9.7GB/s for 10KB messages.

Latency reduced by nearly two orders of magnitude.

Performance improvements also benefited an OMG-compliant avionics DDS.

Abstract

Leveraging one-sided RDMA for applications that replicate small data objects can be surprisingly difficult: such uses amplify any protocol overheads. Spindle is a set of optimization techniques for systematically tackling this class of challenges for atomic multicast over RDMA. These include memory polling optimizations using novel sender and receiver batching techniques, null-message send logic, and improved multi-thread synchronization. We applied Spindle to Derecho, an open-source C++ library for atomic multicast, and obtained significant performance improvements both for the library itself and for an OMG-compliant avionics DDS built over Derecho. Derecho's multicast bandwidth utilization for 10KB messages rose from 1GB/s to 9.7GB/s on a 12.5GB/s network, and it became more robust to delays. Interestingly, although some of our techniques employ batching, latency dropped by nearly two orders of magnitude. Spindle optimizations should also be of value in other RDMA applications limited by the speed of coordination.