Zero-Cost, Arrow-Enabled Data Interface for Apache Spark

TL;DR

This paper introduces a zero-cost data interface between Apache Spark and Arrow, enabling seamless, high-performance interoperability for distributed data processing without additional overhead.

Contribution

The work presents a novel, open-source data interface that separates computation and data layers, improving interoperability and performance between Spark and Arrow-based data sources.

Findings

Data interface is zero-cost or more performant than native Spark

Enables seamless access to Arrow Dataset API data sources from Spark

Separates computation and data layers for better interoperability

Abstract

Distributed data processing ecosystems are widespread and their components are highly specialized, such that efficient interoperability is urgent. Recently, Apache Arrow was chosen by the community to serve as a format mediator, providing efficient in-memory data representation. Arrow enables efficient data movement between data processing and storage engines, significantly improving interoperability and overall performance. In this work, we design a new zero-cost data interoperability layer between Apache Spark and Arrow-based data sources through the Arrow Dataset API. Our novel data interface helps separate the computation (Spark) and data (Arrow) layers. This enables practitioners to seamlessly use Spark to access data from all Arrow Dataset API-enabled data sources and frameworks. To benefit our community, we open-source our work and show that consuming data through Apache Arrow is zero-cost: our novel data interface is either on-par or more performant than native Spark.