Performance and Fault Tolerance in the StoreTorrent Parallel Filesystem

TL;DR

This paper introduces StoreTorrent, a distributed filesystem optimized for small data records, with innovative techniques to improve performance and fault tolerance, enabling efficient analysis of large datasets on commodity hardware.

Contribution

The paper presents new application-OS pipelining and metadata structures, along with peer-to-peer replica-location communication, to significantly enhance small read/write performance and fault tolerance.

Findings

StoreTorrent achieves 1.69 GB/s small write throughput.

Supports network reads at 8.47 GB/s.

Parallel analysis tasks reach full storage node speed.

Abstract

With a goal of supporting the timely and cost-effective analysis of Terabyte datasets on commodity components, we present and evaluate StoreTorrent, a simple distributed filesystem with integrated fault tolerance for efficient handling of small data records. Our contributions include an application-OS pipelining technique and metadata structure to increase small write and read performance by a factor of 1-10, and the use of peer-to-peer communication of replica-location indexes to avoid transferring data during parallel analysis even in a degraded state. We evaluated StoreTorrent, PVFS, and Gluster filesystems using 70 storage nodes and 560 parallel clients on an 8-core/node Ethernet cluster with directly attached SATA disks. StoreTorrent performed parallel small writes at an aggregate rate of 1.69 GB/s, and supported reads over the network at 8.47 GB/s. We ported a parallel analysis task and demonstrate it achieved parallel reads at the full aggregate speed of the storage node local filesystems.