RIO: Order-Preserving and CPU-Efficient Remote Storage Access

TL;DR

Rio introduces an out-of-order, asynchronous I/O pipeline for remote storage that maintains order externally, significantly improving throughput and CPU efficiency over existing systems by leveraging layered design and concurrent device operation.

Contribution

The paper presents Rio, a novel storage architecture that enables out-of-order execution of ordered requests, enhancing performance and efficiency in remote NVMe over RDMA systems.

Findings

Rio outperforms Linux NVMe over RDMA by two orders of magnitude in throughput.

RioFS improves RocksDB throughput by up to 1.9 times.

The approach achieves near orderless request performance while preserving external order.

Abstract

Modern NVMe SSDs and RDMA networks provide dramatically higher bandwidth and concurrency. Existing networked storage systems (e.g., NVMe over Fabrics) fail to fully exploit these new devices due to inefficient storage ordering guarantees. Severe synchronous execution for storage order in these systems stalls the CPU and I/O devices and lowers the CPU and I/O performance efficiency of the storage system. We present Rio, a new approach to the storage order of remote storage access. The key insight in Rio is that the layered design of the software stack, along with the concurrent and asynchronous network and storage devices, makes the storage stack conceptually similar to the CPU pipeline. Inspired by the CPU pipeline that executes out-of-order and commits in-order, Rio introduces the I/O pipeline that allows internal out-of-order and asynchronous execution for ordered write requests while offering intact external storage order to applications. Together with merging consecutive ordered requests, these design decisions make for write throughput and CPU efficiency close to that of orderless requests. We implement Rio in Linux NVMe over RDMA stack, and further build a file system named RioFS atop Rio. Evaluations show that Rio outperforms Linux NVMe over RDMA and a state-of-the-art storage stack named Horae by two orders of magnitude and 4.9 times on average in terms of throughput of ordered write requests, respectively. RioFS increases the throughput of RocksDB by 1.9 times and 1.5 times on average, against Ext4 and HoraeFS, respectively.