PyParSVD: A streaming, distributed and randomized singular-value-decomposition library

TL;DR

PyParSVD is a Python library that provides a scalable, streaming, and randomized approach to singular value decomposition, enabling efficient analysis of large scientific datasets on high-performance computing systems.

Contribution

It introduces a novel Python library implementing a streaming, distributed, and randomized SVD algorithm, suitable for large-scale scientific data analysis.

Findings

Effective extraction of coherent structures from scientific data.

Successful weak scaling on up to 256 nodes of the Theta supercomputer.

Demonstrates potential for large-scale data analysis.

Abstract

We introduce PyParSVD\footnote{https://github.com/Romit-Maulik/PyParSVD}, a Python library that implements a streaming, distributed and randomized algorithm for the singular value decomposition. To demonstrate its effectiveness, we extract coherent structures from scientific data. Futhermore, we show weak scaling assessments on up to 256 nodes of the Theta machine at Argonne Leadership Computing Facility, demonstrating potential for large-scale data analyses of practical data sets.