Efficient GPU implementation of randomized SVD and its applications

TL;DR

This paper presents a GPU-optimized implementation of randomized SVD that leverages parallel matrix operations to significantly reduce computation time, enhancing efficiency in machine learning applications.

Contribution

The work reformulates randomized SVD to incorporate fast matrix multiplication on GPUs, enabling fully parallel processing and outperforming existing methods.

Findings

GPU implementation outperforms traditional CPU methods

Reformulated algorithm exploits BLAS-3 operations for speed

Results integrated into official CUDA library

Abstract

Matrix decompositions are ubiquitous in machine learning, including applications in dimensionality reduction, data compression and deep learning algorithms. Typical solutions for matrix decompositions have polynomial complexity which significantly increases their computational cost and time. In this work, we leverage efficient processing operations that can be run in parallel on modern Graphical Processing Units (GPUs), predominant computing architecture used e.g. in deep learning, to reduce the computational burden of computing matrix decompositions. More specifically, we reformulate the randomized decomposition problem to incorporate fast matrix multiplication operations (BLAS-3) as building blocks. We show that this formulation, combined with fast random number generators, allows to fully exploit the potential of parallel processing implemented in GPUs. Our extensive evaluation confirms the superiority of this approach over the competing methods and we release the results of this research as a part of the official CUDA implementation (https://docs.nvidia.com/cuda/cusolver/index.html).