AcceleratedKernels.jl: Cross-Architecture Parallel Algorithms from a Unified, Transpiled Codebase

TL;DR

AcceleratedKernels.jl is a Julia library that enables cross-architecture parallel computing with a unified codebase, achieving high performance and exceptional composability across diverse hardware including GPUs and CPUs.

Contribution

It introduces a transpilation-based backend-agnostic framework for parallel programming in Julia, simplifying implementation and enabling efficient CPU-GPU co-processing.

Findings

Performance comparable to C and OpenMP implementations.

Achieved world-class sorting throughput of 538-855 GB/s on NVIDIA A100 GPUs.

GPU interconnects like NVLink significantly speed up HPC tasks.

Abstract

AcceleratedKernels.jl is introduced as a backend-agnostic library for parallel computing in Julia, natively targeting NVIDIA, AMD, Intel, and Apple accelerators via a unique transpilation architecture. Written in a unified, compact codebase, it enables productive parallel programming with minimised implementation and usage complexities. Benchmarks of arithmetic-heavy kernels show performance on par with C and OpenMP-multithreaded CPU implementations, with Julia sometimes offering more consistent and predictable numerical performance than conventional C compilers. Exceptional composability is highlighted as simultaneous CPU-GPU co-processing is achievable - such as CPU-GPU co-sorting - with transparent use of hardware-specialised MPI implementations. Tests on the Baskerville Tier 2 UK HPC cluster achieved world-class sorting throughputs of 538-855 GB/s using 200 NVIDIA A100 GPUs, comparable to the highest literature-reported figure of 900 GB/s achieved on 262,144 CPU cores. The use of direct NVLink GPU-to-GPU interconnects resulted in a 4.93x speedup on average; normalised by a combined capital, running and environmental cost, communication-heavy HPC tasks only become economically viable on GPUs if GPUDirect interconnects are employed.