Orderly Management of Packets in RDMA by Eunomia

TL;DR

Eunomia introduces an ordering layer for RDMA networks that handles packet reordering, enabling the use of advanced load balancing and topologies to significantly improve datacenter performance.

Contribution

The paper presents Eunomia, a novel NIC-level ordering layer that relaxes in-order delivery constraints in RDMA, allowing for performance-enhancing techniques to be effectively employed.

Findings

Reduces flow completion times by up to 85% with load balancers.

Enables RDMA in expander topologies like Jellyfish, achieving 60% lower completion times.

Demonstrates FPGA implementation feasibility and large-scale simulation performance.

Abstract

To fulfill the low latency requirements of today's applications, deployment of RDMA in datacenters has become prevalent over the recent years. However, the in-order delivery requirement of RDMAs prevents them from leveraging powerful techniques that help improve the performance of datacenters, ranging from fine-grained load balancers to throughput-optimal expander topologies. We demonstrate experimentally that these techniques significantly deteriorate the performance in an RDMA network because they induce packet reordering. Furthermore, lifting the in-order delivery constraint enhances the flexibility of RDMA networks and enables them to employ these performance-enhancing techniques. To realize this, we propose an ordering layer, Eunomia, to equip RDMA NICs to handle packet reordering. Eunomia employs a hybrid-dynamic bitmap structure that efficiently uses the limited on-chip memory with the help of a customized memory controller and handles high degrees of packet reordering. We evaluate the feasibility of Eunomia through an FPGA-based implementation and its performance through large-scale simulations. We show that Eunomia enables a wide range of applications in RDMA datacenter networks, such as fine-grained load balancers which improve performance by reducing average flow completion times by 85% and 52% compared to ECMP and Conweave, respectively, or employment of RDMA in expander topologies like Jellyfish which allows up to 60% lower flow completion times and higher throughput gains compared to Fat tree.