RDD-Eclat: Approaches to Parallelize Eclat Algorithm on Spark RDD Framework (Extended Version)

TL;DR

This paper introduces RDD-Eclat, a parallel Eclat algorithm on Spark RDD, demonstrating significant performance improvements and scalability over existing methods like Apriori in frequent itemset mining.

Contribution

The paper presents the first parallel Eclat algorithm on Spark RDD with five variants, enhancing efficiency for large-scale frequent itemset mining.

Findings

RDD-Eclat outperforms Spark-based Apriori by many times.

The algorithms are scalable with increasing cores and dataset size.

Experimental results validate the efficiency and scalability of RDD-Eclat.

Abstract

Frequent itemset mining (FIM) is a highly computational and data intensive algorithm. Therefore, parallel and distributed FIM algorithms have been designed to process large volume of data in a reduced time. Recently, a number of FIM algorithms have been designed on Hadoop MapReduce, a distributed big data processing framework. But, due to heavy disk I/O, MapReduce is found to be inefficient for the highly iterative FIM algorithms. Therefore, Spark, a more efficient distributed data processing framework, has been developed with in-memory computation and resilient distributed dataset (RDD) features to support the iterative algorithms. On this framework, Apriori and FP-Growth based FIM algorithms have been designed on the Spark RDD framework, but Eclat-based algorithm has not been explored yet. In this paper, RDD-Eclat, a parallel Eclat algorithm on the Spark RDD framework is proposed with its five variants. The proposed algorithms are evaluated on the various benchmark datasets, and the experimental results show that RDD-Eclat outperforms the Spark-based Apriori by many times. Also, the experimental results show the scalability of the proposed algorithms on increasing the number of cores and size of the dataset.