Performance and Fault Tolerance in the StoreTorrent Parallel Filesystem
Federico D. Sacerdoti

TL;DR
This paper introduces StoreTorrent, a distributed filesystem optimized for small data records, with innovative techniques to improve performance and fault tolerance, enabling efficient analysis of large datasets on commodity hardware.
Contribution
The paper presents new application-OS pipelining and metadata structures, along with peer-to-peer replica-location communication, to significantly enhance small read/write performance and fault tolerance.
Findings
StoreTorrent achieves 1.69 GB/s small write throughput.
Supports network reads at 8.47 GB/s.
Parallel analysis tasks reach full storage node speed.
Abstract
With a goal of supporting the timely and cost-effective analysis of Terabyte datasets on commodity components, we present and evaluate StoreTorrent, a simple distributed filesystem with integrated fault tolerance for efficient handling of small data records. Our contributions include an application-OS pipelining technique and metadata structure to increase small write and read performance by a factor of 1-10, and the use of peer-to-peer communication of replica-location indexes to avoid transferring data during parallel analysis even in a degraded state. We evaluated StoreTorrent, PVFS, and Gluster filesystems using 70 storage nodes and 560 parallel clients on an 8-core/node Ethernet cluster with directly attached SATA disks. StoreTorrent performed parallel small writes at an aggregate rate of 1.69 GB/s, and supported reads over the network at 8.47 GB/s. We ported a parallel analysis…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsAdvanced Data Storage Technologies · Caching and Content Delivery · Peer-to-Peer Network Technologies
