REX: Recursive, Delta-Based Data-Centric Computation

TL;DR

REX introduces a delta-based recursive computation platform that unifies the strengths of DBMSs and cloud platforms, enabling efficient iterative data analysis with significant speedups.

Contribution

The paper presents a novel delta-oriented programming model and runtime system that efficiently supports iterative computations, combining robustness and performance improvements.

Findings

Achieves 2.5 to 100 times speedup over existing methods.

Supports fault tolerance and extensibility in iterative data analysis.

Unifies strengths of DBMSs and cloud platforms for recursive computations.

Abstract

In today's Web and social network environments, query workloads include ad hoc and OLAP queries, as well as iterative algorithms that analyze data relationships (e.g., link analysis, clustering, learning). Modern DBMSs support ad hoc and OLAP queries, but most are not robust enough to scale to large clusters. Conversely, "cloud" platforms like MapReduce execute chains of batch tasks across clusters in a fault tolerant way, but have too much overhead to support ad hoc queries. Moreover, both classes of platform incur significant overhead in executing iterative data analysis algorithms. Most such iterative algorithms repeatedly refine portions of their answers, until some convergence criterion is reached. However, general cloud platforms typically must reprocess all data in each step. DBMSs that support recursive SQL are more efficient in that they propagate only the changes in each step -- but they still accumulate each iteration's state, even if it is no longer useful. User-defined functions are also typically harder to write for DBMSs than for cloud platforms. We seek to unify the strengths of both styles of platforms, with a focus on supporting iterative computations in which changes, in the form of deltas, are propagated from iteration to iteration, and state is efficiently updated in an extensible way. We present a programming model oriented around deltas, describe how we execute and optimize such programs in our REX runtime system, and validate that our platform also handles failures gracefully. We experimentally validate our techniques, and show speedups over the competing methods ranging from 2.5 to nearly 100 times.