RDD-Eclat: Approaches to Parallelize Eclat Algorithm on Spark RDD Framework

TL;DR

This paper introduces RDD-Eclat, a parallel Eclat algorithm optimized for Spark RDD, demonstrating significant performance improvements and scalability over existing algorithms like Apriori on benchmark datasets.

Contribution

The paper presents the first parallel Eclat algorithm on Spark RDD with five variants, enhancing frequent itemset mining efficiency in distributed environments.

Findings

RDD-Eclat outperforms Spark-based Apriori significantly.

The algorithms show good scalability with more cores and larger datasets.

Experimental results confirm the effectiveness of the proposed methods.

Abstract

Initially, a number of frequent itemset mining (FIM) algorithms have been designed on the Hadoop MapReduce, a distributed big data processing framework. But, due to heavy disk I/O, MapReduce is found to be inefficient for such highly iterative algorithms. Therefore, Spark, a more efficient distributed data processing framework, has been developed with in-memory computation and resilient distributed dataset (RDD) features to support the iterative algorithms. On the Spark RDD framework, Apriori and FP-Growth based FIM algorithms have been designed, but Eclat-based algorithm has not been explored yet. In this paper, RDD-Eclat, a parallel Eclat algorithm on the Spark RDD framework is proposed with its five variants. The proposed algorithms are evaluated on the various benchmark datasets, which shows that RDD-Eclat outperforms the Spark-based Apriori by many times. Also, the experimental results show the scalability of the proposed algorithms on increasing the number of cores and size of the dataset.