SHIFT: Exploring the Boundary of RDMA Network Fault Tolerance

TL;DR

This paper introduces SHIFT, a user-space RDMA layer that extends fault tolerance across NICs in distributed training, overcoming fundamental limitations and maintaining training progress despite network failures.

Contribution

We demonstrate the fundamental Trilemma in cross-NIC RDMA failover and propose SHIFT, a novel solution that provides fault tolerance while preserving memory semantics.

Findings

SHIFT incurs negligible overhead during normal operation.

SHIFT successfully masks NIC failures and link anomalies.

Training continues without costly restarts despite network faults.

Abstract

Under gang scheduling for large-scale distributed large language model (LLM) training, a single network anomaly can stall or abort an entire job. Current network fault tolerance mechanisms typically adopt a ``fallback and bypass'' approach within the switching fabric and at the access layer, tolerating in-network and access-layer failures. We explore whether RDMA fault tolerance can be extended to the cross-NIC level by failing over traffic to intra-host backup NICs. For the first time, we prove a fundamental Trilemma: it is impossible to have Cross-NIC RDMA failover that simultaneously preserves Exactly-Once Execution, Receiver-NIC Opacity, and a Zero-Copy datapath. Fortunately, we observe that dominant training frameworks (e.g., NCCL) rely on idempotent bulk transfers that tolerate relaxed memory ordering, as long as notification ordering is preserved. Leveraging this insight, we present SHIFT, a user-space RDMA layer that provides cross-NIC fault tolerance while preserving correct memory semantics. We implement SHIFT in \texttt{rdma-core} and evaluate it with PyTorch distributed training. Results show that SHIFT incurs negligible overhead during normal operation and successfully masks fatal NIC failures and link anomalies, allowing training to continue without costly restarts.