Helping HPC Users Specify Job Memory Requirements via Machine Learning

TL;DR

This paper presents a machine learning-based tool integrated with HPC batch schedulers to accurately predict user job memory requirements, improving resource allocation and user planning.

Contribution

The paper introduces a novel machine learning approach for predicting HPC job memory needs and demonstrates its integration with batch scheduler systems using real production data.

Findings

Accurate memory prediction improves resource utilization.

The tool effectively integrates with existing batch schedulers.

Validation on real systems shows promising results.

Abstract

Resource allocation in High Performance Computing (HPC) settings is still not easy for end-users due to the wide variety of application and environment configuration options. Users have difficulties to estimate the number of processors and amount of memory required by their jobs, select the queue and partition, and estimate when job output will be available to plan for next experiments. Apart from wasting infrastructure resources by making wrong allocation decisions, overall user response time can also be negatively impacted. Techniques that exploit batch scheduler systems to predict waiting time and runtime of user jobs have already been proposed. However, we observed that such techniques are not suitable for predicting job memory usage. In this paper we introduce a tool to help users predict their memory requirements using machine learning. We describe the integration of the tool with a batch scheduler system, discuss how batch scheduler log data can be exploited to generate memory usage predictions through machine learning, and present results of two production systems containing thousands of jobs.