SparkCL: A Unified Programming Framework for Accelerators on Heterogeneous Clusters

TL;DR

SparkCL is a unified Java-based framework that integrates heterogeneous accelerators like FPGAs and GPUs into Apache Spark, enabling seamless offloading and programming of diverse compute devices within big data workflows.

Contribution

It introduces a novel unified programming framework that supports multiple accelerator types in Spark, simplifying heterogeneous computing in big data environments.

Findings

Supports multiple accelerator types via OpenCL in Spark

Enables offloading computations to accelerators seamlessly

Provides a single code base for diverse hardware targets

Abstract

We introduce SparkCL, an open source unified programming framework based on Java, OpenCL and the Apache Spark framework. The motivation behind this work is to bring unconventional compute cores such as FPGAs/GPUs/APUs/DSPs and future core types into mainstream programming use. The framework allows equal treatment of different computing devices under the Spark framework and introduces the ability to offload computations to acceleration devices. The new framework is seamlessly integrated into the standard Spark framework via a Java-OpenCL device programming layer which is based on Aparapi and a Spark programming layer that includes new kernel function types and modified Spark transformations and actions. The framework allows a single code base to target any type of compute core that supports OpenCL and easy integration of new core types into a Spark cluster.