Distributed Function Minimization in Apache Spark

TL;DR

This paper presents an open-source distributed function minimization implementation built on Apache Spark, utilizing gradient and quasi-Newton methods, demonstrated through Optimal Transport and scalability testing on classification and regression tasks.

Contribution

It introduces a novel distributed minimization tool on Spark with applications to Optimal Transport and scalability analysis.

Findings

Successful implementation of distributed minimization methods.

Effective scalability on classification and regression problems.

Application to Optimal Transport demonstrates versatility.

Abstract

We report on an open-source implementation for distributed function minimization on top of Apache Spark by using gradient and quasi-Newton methods. We show-case it with an application to Optimal Transport and some scalability tests on classification and regression problems.