Adaptive filter ordering in Spark

TL;DR

This paper introduces an adaptive method for reordering predicates in Spark data streams, improving execution efficiency by dynamically adjusting to changing data statistics, with an open-source prototype demonstrating its feasibility.

Contribution

It presents a novel engineering approach to make Spark's execution engine adaptive through predicate reordering based on evolving data statistics.

Findings

System extension is available as open-source.

Experimental results show manageable overhead.

Sensitivity to tuning parameters is observed.

Abstract

This report describes a technical methodology to render the Apache Spark execution engine adaptive. It presents the engineering solutions, which specifically target to adaptively reorder predicates in data streams with evolving statistics. The system extension developed is available as an open-source prototype. Indicative experimental results show its overhead and sensitivity to tuning parameters.