HPC-LLM: Practical Domain Adaptation and Retrieval-Augmented Generation for HPC Support

TL;DR

HPC-LLM is a domain-specific, retrieval-augmented language model fine-tuned for HPC workflows, achieving high performance with lower resource needs compared to larger general models.

Contribution

The paper introduces HPC-LLM, a domain-adapted LLM for HPC support, utilizing retrieval augmentation and lightweight fine-tuning to improve operational assistance.

Findings

The adapted 8B model performs comparably to larger models like Qwen 2.5 14B.

HPC-LLM operates with significantly lower GPU memory and inference latency.

Constructed a large HPC-focused dataset from documentation and synthetic data.

Abstract

Modern scientific research increasingly depends on High-Performance Computing (HPC) infrastructures, yet many researchers face significant operational barriers when interacting with cluster environments, job schedulers, GPU resources, and parallel computing frameworks. General-purpose large language models (LLMs) provide useful coding assistance but often lack the domain-specific operational knowledge required for reliable HPC support. This paper presents HPC-LLM, a retrieval augmented and domain-adapted assistant designed to support common HPC workflows including Slurm scheduling, MPI execution, GPU utilization, filesystem management, and cluster troubleshooting. The proposed framework integrates automated documentation ingestion, dense retrieval, lightweight domain adaptation using QLoRA, and local inference within a modular orchestration pipeline. To support domain adaptation, we construct an HPC-oriented corpus from publicly available university HPC documentation, curated operational examples, and synthetic instruction-answer pairs generated from retrieved HPC content. The resulting dataset contains approximately 9,000 to 24,000 HPC-focused training examples spanning job scheduling, GPU computing, distributed training, storage systems, and cluster administration topics. We fine-tune Llama 3.1 8B using QLoRA and evaluate the resulting model against several open weight baselines under retrieval-augmented settings on JetStream2 infrastructure. Experimental results indicate that the adapted 8B model achieves performance comparable to substantially larger general-purpose models while operating under significantly lower GPU memory requirements and inference latency. In particular, the adapted model approaches the performance of Qwen 2.5 14B while requiring substantially fewer computational resources.