JobPruner: A Machine Learning Assistant for Exploring Parameter Spaces in HPC Applications

TL;DR

JobPruner is a machine learning tool that helps HPC users efficiently explore parameter spaces by identifying and eliminating unnecessary jobs, significantly reducing computational costs while maintaining experiment quality.

Contribution

The paper introduces JobPruner, a novel machine learning-based system that automates job elimination in HPC experiments, reducing execution time and resource usage.

Findings

Reduced 93% of jobs in a single experiment

Improved experiment quality in most scenarios

Effective even with low correlation between past and current experiments

Abstract

High Performance Computing (HPC) applications are essential for scientists and engineers to create and understand models and their properties. These professionals depend on the execution of large sets of computational jobs that explore combinations of parameter values. Avoiding the execution of unnecessary jobs brings not only speed to these experiments, but also reductions in infrastructure usage---particularly important due to the shift of these applications to HPC cloud platforms. Our hypothesis is that data generated by these experiments can help users in identifying such jobs. To address this hypothesis we need to understand the similarity levels among multiple experiments necessary for job elimination decisions and the steps required to automate this process. In this paper we present a study and a machine learning-based tool called JobPruner to support parameter exploration in HPC experiments. The tool was evaluated with three real-world use cases from different domains including seismic analysis and agronomy. We observed the tool reduced 93% of jobs in a single experiment, while improving quality in most scenarios. In addition, reduction in job executions was possible even considering past experiments with low correlations.