Performance Optimization of MapReduce-based Apriori Algorithm on Hadoop Cluster

TL;DR

This paper introduces improved MapReduce-based Apriori algorithms that optimize performance by combining multiple passes and selectively skipping pruning, resulting in faster execution on Hadoop clusters.

Contribution

It proposes VFPC and ETDPC algorithms with optimized versions that enhance efficiency and robustness over existing combined-counting methods.

Findings

Optimized algorithms reduce execution time significantly.

Skipping pruning in some passes decreases overall computation.

Proposed methods outperform traditional approaches in experiments.

Abstract

Many techniques have been proposed to implement the Apriori algorithm on MapReduce framework but only a few have focused on performance improvement. FPC (Fixed Passes Combined-counting) and DPC (Dynamic Passes Combined-counting) algorithms combine multiple passes of Apriori in a single MapReduce phase to reduce the execution time. In this paper, we propose improved MapReduce based Apriori algorithms VFPC (Variable Size based Fixed Passes Combined-counting) and ETDPC (Elapsed Time based Dynamic Passes Combined-counting) over FPC and DPC. Further, we optimize the multi-pass phases of these algorithms by skipping pruning step in some passes, and propose Optimized-VFPC and Optimized-ETDPC algorithms. Quantitative analysis reveals that counting cost of additional un-pruned candidates produced due to skipped-pruning is less significant than reduction in computation cost due to the same. Experimental results show that VFPC and ETDPC are more robust and flexible than FPC and DPC whereas their optimized versions are more efficient in terms of execution time.