The Landscape and Challenges of HPC Research and LLMs

TL;DR

This paper explores how large language models and high-performance computing can be integrated, emphasizing the potential benefits and adaptations needed for HPC tasks, based on recent advances in LLMs and exascale computing.

Contribution

It proposes adapting LLM techniques for HPC applications and discusses how existing ideas can be improved for high-performance computing tasks.

Findings

LLMs have revolutionized NLP and coding tasks.

HPC has achieved exascale performance levels.

Adapting LLMs for HPC can enhance computational efficiency.

Abstract

Recently, language models (LMs), especially large language models (LLMs), have revolutionized the field of deep learning. Both encoder-decoder models and prompt-based techniques have shown immense potential for natural language processing and code-based tasks. Over the past several years, many research labs and institutions have invested heavily in high-performance computing, approaching or breaching exascale performance levels. In this paper, we posit that adapting and utilizing such language model-based techniques for tasks in high-performance computing (HPC) would be very beneficial. This study presents our reasoning behind the aforementioned position and highlights how existing ideas can be improved and adapted for HPC tasks.