Building Blocks for Network-Accelerated Distributed File Systems

TL;DR

This paper demonstrates how offloading storage policies to programmable SmartNICs significantly improves latency and throughput in distributed file systems, surpassing traditional CPU and RDMA-based methods.

Contribution

It introduces a novel approach of offloading storage policies to programmable SmartNICs using PsPIN, achieving substantial latency improvements in distributed storage operations.

Findings

Latency for writes improved up to 2x

Data replication latency improved up to 2x

Erasure coding latency improved up to 2x

Abstract

High-performance clusters and datacenters pose increasingly demanding requirements on storage systems. If these systems do not operate at scale, applications are doomed to become I/O bound and waste compute cycles. To accelerate the data path to remote storage nodes, remote direct memory access (RDMA) has been embraced by storage systems to let data flow from the network to storage targets, reducing overall latency and CPU utilization. Yet, this approach still involves CPUs on the data path to enforce storage policies such as authentication, replication, and erasure coding. We show how storage policies can be offloaded to fully programmable SmartNICs, without involving host CPUs. By using PsPIN, an open-hardware SmartNIC, we show latency improvements for writes (up to 2x), data replication (up to 2x), and erasure coding (up to 2x), when compared to respective CPU- and RDMA-based alternatives.