Practical Parallel External Memory Algorithms via Simulation of Parallel Algorithms
David E. Robillard

TL;DR
PEMS2 is an improved system that efficiently simulates parallel BSP algorithms in external memory, enabling large-scale data processing with reduced overhead and support for modern multi-core architectures.
Contribution
PEMS2 introduces refinements and new features to reduce simulation overhead and support multi-core processors, improving the practicality of external memory BSP algorithm simulation.
Findings
Significantly improved simulation runtime
Reduced disk space requirements
Enhanced support for multi-core processors
Abstract
This thesis introduces PEMS2, an improvement to PEMS (Parallel External Memory System). PEMS executes Bulk-Synchronous Parallel (BSP) algorithms in an External Memory (EM) context, enabling computation with very large data sets which exceed the size of main memory. Many parallel algorithms have been designed and implemented for Bulk-Synchronous Parallel models of computation. Such algorithms generally assume that the entire data set is stored in main memory at once. PEMS overcomes this limitation without requiring any modification to the algorithm by using disk space as memory for additional "virtual processors". Previous work has shown this to be a promising approach which scales well as computational resources (i.e. processors and disks) are added. However, the technique incurs significant overhead when compared with purpose-built EM algorithms. PEMS2 introduces refinements to the…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsParallel Computing and Optimization Techniques · Advanced Data Storage Technologies · Distributed and Parallel Computing Systems
