Out-of-core singular value decomposition

TL;DR

This paper presents a scalable, out-of-core randomized SVD algorithm capable of efficiently factorizing very large matrices that exceed main memory, with features like parallelization, load balancing, and resumability.

Contribution

It introduces an innovative out-of-core randomized SVD method that handles large matrices efficiently, supporting parallel processing and interruption recovery.

Findings

Handles arbitrarily large dense and sparse matrices

Supports parallel and out-of-core processing

Enables resumption of interrupted computations

Abstract

Singular value decomposition (SVD) is a standard matrix factorization technique that produces optimal low-rank approximations of matrices. It has diverse applications, including machine learning, data science and signal processing. However, many common problems involve very large matrices that cannot fit in the main memory of commodity computers, making it impractical to use standard SVD algorithms that assume fast random access or large amounts of space for intermediate calculations. To address this issue, we have implemented an out-of-core (external memory) randomized SVD solution that is fully scalable and efficiently parallelizable. This solution factors both dense and sparse matrices of arbitrarily large size within arbitrarily small memory limits, efficiently using out-of-core storage as needed. It uses an innovative technique for partitioning matrices that lends itself to out-of-core and parallel processing, as well as memory and I/O use planning, automatic load balancing, performance tuning, and makes possible a number of other practical enhancements to the current state-of-the-art. Furthermore, by using persistent external storage (generally HDDs or SSDs), users can resume interrupted operations without having to recalculate previously performed steps, solving a major practical problem in factoring very large matrices.