DynIMS: A Dynamic Memory Controller for In-memory Storage on HPC Systems
Pengfei Xuan, Feng Luo, Rong Ge, Pradip K Srimani

TL;DR
DynIMS is a dynamic memory controller that adaptively manages in-memory storage in HPC systems, significantly improving performance by optimizing memory allocation for compute and data-intensive tasks.
Contribution
This paper introduces DynIMS, a novel feedback-based control system that dynamically adjusts in-memory storage capacity based on real-time memory demands in HPC environments.
Findings
Up to 5X performance improvement with DynIMS
Effective online inference of memory demands
Enhanced resource utilization in HPC systems
Abstract
In order to boost the performance of data-intensive computing on HPC systems, in-memory computing frameworks, such as Apache Spark and Flink, use local DRAM for data storage. Optimizing the memory allocation to data storage is critical to delivering performance to traditional HPC compute jobs and throughput to data-intensive applications sharing the HPC resources. Current practices that statically configure in-memory storage may leave inadequate space for compute jobs or lose the opportunity to utilize more available space for data-intensive applications. In this paper, we explore techniques to dynamically adjust in-memory storage and make the right amount of space for compute jobs. We have developed a dynamic memory controller, DynIMS, which infers memory demands of compute tasks online and employs a feedback-based control model to adapt the capacity of in-memory storage. We test…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsAdvanced Data Storage Technologies · Parallel Computing and Optimization Techniques · Distributed and Parallel Computing Systems
