REPS: Recycled Entropy Packet Spraying for Adaptive Load Balancing and Failure Mitigation

TL;DR

REPS is a lightweight, decentralized adaptive load balancing algorithm for datacenter networks that quickly reroutes traffic during failures, improving utilization and resilience with minimal state overhead.

Contribution

It introduces REPS, a novel per-packet adaptive load balancing method that rapidly recovers from failures and requires minimal per-connection state, suitable for next-gen Ethernet networks.

Findings

REPS achieves rerouting in less than 100 microseconds.

REPS uses less than 25 bytes of per-connection state.

REPS outperforms existing solutions in large-scale simulations and FPGA tests.

Abstract

Next-generation datacenters require highly efficient network load balancing to manage the growing scale of artificial intelligence (AI) training and general datacenter traffic. However, existing Ethernet-based solutions, such as Equal Cost Multi-Path (ECMP) and oblivious packet spraying (OPS), struggle to maintain high network utilization due to both increasing traffic demands and the expanding scale of datacenter topologies, which also exacerbate network failures. To address these limitations, we propose REPS, a lightweight decentralized per-packet adaptive load balancing algorithm designed to optimize network utilization while ensuring rapid recovery from link failures. REPS adapts to network conditions by caching good-performing paths. In case of a network failure, REPS re-routes traffic away from it in less than 100 microseconds. REPS is designed to be deployed with next-generation out-of-order transports, such as Ultra Ethernet, and uses less than 25 bytes of per-connection state regardless of the topology size. We extensively evaluate REPS in large-scale simulations and FPGA-based NICs.