Efficient Multi-way Theta-Join Processing Using MapReduce

TL;DR

This paper presents a cost-effective method for processing multi-way Theta-join queries efficiently using MapReduce, overcoming limitations of traditional approaches in distributed computing environments for large-scale data analysis.

Contribution

It introduces a novel cost model and execution strategy for multi-way Theta-joins in MapReduce, enabling efficient query processing with minimized total processing time.

Findings

Significant improvement over existing query evaluation strategies.

Effective execution of chain-typed Theta-joins with a single MapReduce job.

Enhanced support for complex join queries in distributed data processing.

Abstract

Multi-way Theta-join queries are powerful in describing complex relations and therefore widely employed in real practices. However, existing solutions from traditional distributed and parallel databases for multi-way Theta-join queries cannot be easily extended to fit a shared-nothing distributed computing paradigm, which is proven to be able to support OLAP applications over immense data volumes. In this work, we study the problem of efficient processing of multi-way Theta-join queries using MapReduce from a cost-effective perspective. Although there have been some works using the (key,value) pair-based programming model to support join operations, efficient processing of multi-way Theta-join queries has never been fully explored. The substantial challenge lies in, given a number of processing units (that can run Map or Reduce tasks), mapping a multi-way Theta-join query to a number of MapReduce jobs and having them executed in a well scheduled sequence, such that the total processing time span is minimized. Our solution mainly includes two parts: 1) cost metrics for both single MapReduce job and a number of MapReduce jobs executed in a certain order; 2) the efficient execution of a chain-typed Theta-join with only one MapReduce job. Comparing with the query evaluation strategy proposed in [23] and the widely adopted Pig Latin and Hive SQL solutions, our method achieves significant improvement of the join processing efficiency.