Matrix Computations and Optimization in Apache Spark

TL;DR

This paper presents methods for efficient distributed matrix computations and optimization routines in Apache Spark, enabling scalable linear algebra and convex programming on cluster systems with practical benchmarks.

Contribution

It introduces a framework for distributed matrix operations in Spark, including SVD and convex optimization, with implementations integrated into Spark and supported commercially.

Findings

Distributed matrix operations outperform single-node counterparts.

The SVD implementation exploits cluster computing for large-scale data.

Benchmark results demonstrate the efficiency of JVM-based matrix computations.

Abstract

We describe matrix computations available in the cluster programming framework, Apache Spark. Out of the box, Spark provides abstractions and implementations for distributed matrices and optimization routines using these matrices. When translating single-node algorithms to run on a distributed cluster, we observe that often a simple idea is enough: separating matrix operations from vector operations and shipping the matrix operations to be ran on the cluster, while keeping vector operations local to the driver. In the case of the Singular Value Decomposition, by taking this idea to an extreme, we are able to exploit the computational power of a cluster, while running code written decades ago for a single core. Another example is our Spark port of the popular TFOCS optimization package, originally built for MATLAB, which allows for solving Linear programs as well as a variety of other convex programs. We conclude with a comprehensive set of benchmarks for hardware accelerated matrix computations from the JVM, which is interesting in its own right, as many cluster programming frameworks use the JVM. The contributions described in this paper are already merged into Apache Spark and available on Spark installations by default, and commercially supported by a slew of companies which provide further services.