MLlib: Machine Learning in Apache Spark

TL;DR

MLlib is a comprehensive distributed machine learning library integrated with Apache Spark, offering scalable algorithms, primitives, and APIs to facilitate large-scale data analysis and model development.

Contribution

This paper introduces MLlib, a new open-source machine learning library for Spark, with efficient algorithms, primitives, and APIs for large-scale data processing.

Findings

MLlib supports a wide range of learning algorithms.

MLlib has a rapidly growing open-source community.

MLlib simplifies development of end-to-end machine learning pipelines.

Abstract

Apache Spark is a popular open-source platform for large-scale data processing that is well-suited for iterative machine learning tasks. In this paper we present MLlib, Spark's open-source distributed machine learning library. MLlib provides efficient functionality for a wide range of learning settings and includes several underlying statistical, optimization, and linear algebra primitives. Shipped with Spark, MLlib supports several languages and provides a high-level API that leverages Spark's rich ecosystem to simplify the development of end-to-end machine learning pipelines. MLlib has experienced a rapid growth due to its vibrant open-source community of over 140 contributors, and includes extensive documentation to support further growth and to let users quickly get up to speed.