Performant Unified GPU Kernels for Portable Singular Value Computation Across Hardware and Precision

TL;DR

This paper introduces a portable, GPU-accelerated SVD implementation in Julia that supports diverse hardware and data types, including Apple Metal GPUs and half precision, achieving high performance across platforms.

Contribution

It presents the first GPU-accelerated SVD supporting Apple Metal GPUs and half precision, with a unified, hardware-agnostic implementation in Julia.

Findings

Outperforms most linear algebra libraries for large matrices

Supports diverse GPU architectures and data types

Achieves 80%-90% of cuSOLVER performance on large matrices

Abstract

This paper presents a portable, GPU-accelerated implementation of a QR-based singular value computation algorithm in Julia. The singular value ecomposition (SVD) is a fundamental numerical tool in scientific computing and machine learning, providing optimal low-rank matrix approximations. Its importance has increased even more in large-scale machine learning pipelines, including large language models (LLMs), where it enables low-rank adaptation (LoRA). The implemented algorithm is based on the classic two-stage QR reduction, consisting of successive matrix reduction to band form and bidiagonal form. Our implementation leverages Julia's multiple dispatch and metaprogramming capabilities, integrating with the GPUArrays and KernelAbstractions frameworks to provide a unified type and hardware-agnostic function. It supports diverse GPU architectures and data types, and is, to our knowledge, the first GPU-accelerated singular value implementation to support Apple Metal GPUs and half precision. Performance results on multiple GPU backends and data types demonstrate that portability does not require sacrificing performance: the unified function outperforms most linear algebra libraries (MAGMA, SLATE, rocSOLVER, oneMKL) for matrix sizes larger than 1024x1024, and achieves 80%-90% of the performance of cuSOLVER for large matrices.