Identifying the potential of Near Data Computing for Apache Spark

TL;DR

This paper explores the potential of Near Data Computing architectures, specifically hybrid processing-in-memory and in-storage processing, to enhance the performance of Apache Spark for big data analytics.

Contribution

It hypothesizes and investigates the benefits of NDC architectures for Apache Spark through extensive profiling on Ivy Bridge servers.

Findings

NDC architectures show promise for improving Spark performance

Profiling indicates potential efficiency gains with NDC

Hybrid processing-in-memory and in-storage approaches are beneficial

Abstract

While cluster computing frameworks are continuously evolving to provide real-time data analysis capabilities, Apache Spark has managed to be at the forefront of big data analytics for being a unified framework for both, batch and stream data processing. There is also a renewed interest is Near Data Computing (NDC) due to technological advancement in the last decade. However, it is not known if NDC architectures can improve the performance of big data processing frameworks such as Apache Spark. In this position paper, we hypothesize in favour of NDC architecture comprising programmable logic based hybrid 2D integrated processing-in-memory and in-storage processing for Apache Spark, by extensive profiling of Apache Spark based workloads on Ivy Bridge Server.