A Constraint Programming-based Job Dispatcher for Modern HPC Systems and Applications

TL;DR

This paper introduces a new constraint programming-based online job dispatcher for modern HPC systems that scales independently of system size, improving dispatching performance in large and complex environments.

Contribution

It presents a novel CP-based dispatcher that models the entire problem independently of system size, enhancing scalability and efficiency in HPC job scheduling.

Findings

Dispatching performance increases significantly in large systems.

Performance improves in systems with complex resource allocation.

Model size remains independent of system size.

Abstract

Constraint Programming (CP) is a well-established area in AI as a programming paradigm for modelling and solving discrete optimization problems, and it has been been successfully applied to tackle the on-line job dispatching problem in HPC systems including those running modern applications. The limitations of the available CP-based job dispatchers may hinder their practical use in today's systems that are becoming larger in size and more demanding in resource allocation. In an attempt to bring basic AI research closer to a deployed application, we present a new CP-based on-line job dispatcher for modern HPC systems and applications. Unlike its predecessors, our new dispatcher tackles the entire problem in CP and its model size is independent of the system size. Experimental results based on a simulation study show that with our approach dispatching performance increases significantly in a large system and in a system where allocation is nontrivial.