Performance Measurements of Supercomputing and Cloud Storage Solutions
Michael Jones, Jeremy Kepner, William Arcand, David Bestor, Bill, Bergeron, Vijay Gadepally, Michael Houle, Matthew Hubbell, Peter Michaleas,, Andrew Prout, Albert Reuther, Siddharth Samsi, Paul Monticiollo

TL;DR
This paper compares the performance of supercomputing Lustre and cloud-based Amazon S3 storage solutions across various parallel I/O scenarios, providing insights for selecting appropriate storage based on workload and environment.
Contribution
It offers the first comprehensive performance measurements of Lustre and Amazon S3 under diverse parallel I/O workloads, guiding storage choice decisions.
Findings
Lustre achieves up to 0.35 terabits/sec on high-end hardware.
Amazon S3 reaches 10 gigabits/sec with parallel I/O.
Parallel I/O utilization enables full network bandwidth performance.
Abstract
Increasing amounts of data from varied sources, particularly in the fields of machine learning and graph analytics, are causing storage requirements to grow rapidly. A variety of technologies exist for storing and sharing these data, ranging from parallel file systems used by supercomputers to distributed block storage systems found in clouds. Relatively few comparative measurements exist to inform decisions about which storage systems are best suited for particular tasks. This work provides these measurements for two of the most popular storage technologies: Lustre and Amazon S3. Lustre is an open-source, high performance, parallel file system used by many of the largest supercomputers in the world. Amazon's Simple Storage Service, or S3, is part of the Amazon Web Services offering, and offers a scalable, distributed option to store and retrieve data from anywhere on the Internet.…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
