Big Data analytics. Three use cases with R, Python and Spark

TL;DR

This paper compares the performance of R, Python, and Spark in handling big data for three use cases, highlighting Spark's efficiency in data processing but limitations in traditional machine learning methods.

Contribution

It provides a comparative analysis of R, Python, and Spark MLlib for big data analytics across three practical applications, illustrating current strengths and limitations.

Findings

Spark excels in data munging and collaborative filtering.

Traditional ML methods in Spark MLlib underperform compared to R and Python.

Conventional learning algorithms in Spark are less competitive in current implementations.

Abstract

Management and analysis of big data are systematically associated with a data distributed architecture in the Hadoop and now Spark frameworks. This article offers an introduction for statisticians to these technologies by comparing the performance obtained by the direct use of three reference environments: R, Python Scikit-learn, Spark MLlib on three public use cases: character recognition, recommending films, categorizing products. As main result, it appears that, if Spark is very efficient for data munging and recommendation by collaborative filtering (non-negative factorization), current implementations of conventional learning methods (logistic regression, random forests) in MLlib or SparkML do not ou poorly compete habitual use of these methods (R, Python Scikit-learn) in an integrated or undistributed architecture