HPC-Coder: Modeling Parallel Programs using Large Language Models

TL;DR

HPC-Coder leverages large language models fine-tuned on HPC code to automate complex tasks like code completion, parallelization, and performance modeling, improving developer productivity in high-performance computing environments.

Contribution

The paper introduces HPC-Coder, a new LLM fine-tuned on HPC code, demonstrating its effectiveness in automating HPC-specific programming tasks.

Findings

HPC-Coder outperforms generic models in auto-completing HPC functions.

It can decorate loops with OpenMP pragmas accurately.

The model effectively predicts performance changes in scientific codes.

Abstract

Parallel programs in high performance computing (HPC) continue to grow in complexity and scale in the exascale era. The diversity in hardware and parallel programming models make developing, optimizing, and maintaining parallel software even more burdensome for developers. One way to alleviate some of these burdens is with automated development and analysis tools. Such tools can perform complex and/or remedial tasks for developers that increase their productivity and decrease the chance for error. Until recently, such tools for code development and performance analysis have been limited in the complexity of tasks they can perform, especially for parallel programs. However, with recent advancements in language modeling, and the availability of large amounts of open-source code related data, these tools have started to utilize predictive language models to automate more complex tasks. In this paper, we show how large language models (LLMs) can be applied to tasks specific to high performance and scientific codes. We introduce a new dataset of HPC and scientific codes and use it to fine-tune several pre-trained models. We compare several pre-trained LLMs on HPC-related tasks and introduce a new model, HPC-Coder, fine-tuned on parallel codes. In our experiments, we show that this model can auto-complete HPC functions where generic models cannot, decorate for loops with OpenMP pragmas, and model performance changes in scientific application repositories as well as programming competition solutions.