Evaluating Large Language Models for Workload Mapping and Scheduling in Heterogeneous HPC Systems

TL;DR

This study evaluates 21 large language models on a complex HPC workload scheduling task, revealing their current capabilities and limitations in reasoning, optimization, and interpretability for structured decision-making.

Contribution

It provides a comprehensive assessment of LLMs' ability to perform constraint-based scheduling from natural language, highlighting their potential as explainable decision-support tools.

Findings

3 models exactly matched the optimal schedule

12 models achieved near-optimal results within 2 minutes

Most models maintained feasible mappings but only half strictly adhered to constraints

Abstract

Large language models (LLMs) are increasingly explored for their reasoning capabilities, yet their ability to perform structured, constraint-based optimization from natural language remains insufficiently understood. This study evaluates twenty-one publicly available LLMs on a representative heterogeneous high-performance computing (HPC) workload mapping and scheduling problem. Each model received the same textual description of system nodes, task requirements, and scheduling constraints, and was required to assign tasks to nodes, compute the total makespan, and explain its reasoning. A manually derived analytical optimum of nine hours and twenty seconds served as the ground truth reference. Three models exactly reproduced the analytical optimum while satisfying all constraints, twelve achieved near-optimal results within two minutes of the reference, and six produced suboptimal schedules with arithmetic or dependency errors. All models generated feasible task-to-node mappings, though only about half maintained strict constraint adherence. Nineteen models produced partially executable verification code, and eighteen provided coherent step-by-step reasoning, demonstrating strong interpretability even when logical errors occurred. Overall, the results define the current capability boundary of LLM reasoning in combinatorial optimization: leading models can reconstruct optimal schedules directly from natural language, but most still struggle with precise timing, data transfer arithmetic, and dependency enforcement. These findings highlight the potential of LLMs as explainable co-pilots for optimization and decision-support tasks rather than autonomous solvers.