A Survey of Singular Value Decomposition Methods for Distributed Tall/Skinny Data

TL;DR

This survey reviews three algorithms for computing the SVD of large, tall/skinny matrices in distributed environments, highlighting their performance on modern supercomputers and discussing future directions.

Contribution

It provides a comprehensive comparison of distributed SVD algorithms for tall/skinny data, including performance results on CPU and GPU architectures.

Findings

Algorithms perform efficiently on Summit supercomputer

GPU implementations show significant speedups

Discussion of alternative approaches for large-scale SVD

Abstract

The Singular Value Decomposition (SVD) is one of the most important matrix factorizations, enjoying a wide variety of applications across numerous application domains. In statistics and data analysis, the common applications of SVD such as Principal Components Analysis (PCA) and linear regression. Usually these applications arise on data that has far more rows than columns, so-called "tall/skinny" matrices. In the big data analytics context, this may take the form of hundreds of millions to billions of rows with only a few hundred columns. There is a need, therefore, for fast, accurate, and scalable tall/skinny SVD implementations which can fully utilize modern computing resources. To that end, we present a survey of three different algorithms for computing the SVD for these kinds of tall/skinny data layouts using MPI for communication. We contextualize these with common big data analytics techniques, principally PCA. Finally, we present both CPU and GPU timing results from the Summit supercomputer, and discuss possible alternative approaches.