ParEval-Repo: A Benchmark Suite for Evaluating LLMs with Repository-level HPC Translation Tasks

TL;DR

This paper introduces ParEval-Repo, a benchmark suite for evaluating large language models' ability to automatically translate entire GPGPU codebases across different execution models, highlighting current capabilities and challenges.

Contribution

The paper presents a novel benchmark and testing framework, ParEval-Repo, for assessing LLMs in translating complex, repository-level scientific computing codebases across GPGPU programming models.

Findings

LLMs can effectively translate small scientific programs.

Scaling to larger codebases remains challenging due to build system and dependency issues.

Translation cost correlates with code complexity.

Abstract

GPGPU architectures have become significantly more diverse in recent years, which has led to an emergence of a variety of specialized programming models and software stacks to support them. Portable programming models exist, but they require significant developer effort to port to and optimize for different hardware architectures. Large language models (LLMs) may help to reduce this programmer burden. In this paper, we present a novel benchmark and testing framework, ParEval-Repo, which can be used to evaluate the efficacy of LLM-based approaches in automatically translating entire codebases across GPGPU execution models. ParEval-Repo includes several scientific computing and AI mini-applications in a range of programming models and levels of repository complexity. We use ParEval-Repo to evaluate a range of state-of-the-art open-source and commercial LLMs, with both a non-agentic and a top-down agentic approach. We assess code generated by the LLMs and approaches in terms of compilability, functional correctness, categories of build errors, and the cost of translation in terms of the number of inference tokens. Our results demonstrate that LLM translation of scientific applications is feasible for small programs but difficulty with generating functional build systems and cross-file dependencies pose challenges in scaling to larger codebases.