Performance Analysis of Apriori Algorithm with Different Data Structures on Hadoop Cluster

TL;DR

This paper evaluates the performance of three data structures—hash tree, trie, and hash table trie—for the Apriori algorithm implemented on Hadoop MapReduce, demonstrating that hash table trie offers superior efficiency in big data mining tasks.

Contribution

It introduces and compares three data structure variations of Apriori on Hadoop MapReduce, highlighting the superior performance of hash table trie for large-scale frequent itemset mining.

Findings

Hash table trie outperforms trie and hash tree in execution time.

Hash tree performance degrades significantly on big datasets.

Experimental results confirm the efficiency of hash table trie in distributed data mining.

Abstract

Mining frequent itemsets from massive datasets is always being a most important problem of data mining. Apriori is the most popular and simplest algorithm for frequent itemset mining. To enhance the efficiency and scalability of Apriori, a number of algorithms have been proposed addressing the design of efficient data structures, minimizing database scan and parallel and distributed processing. MapReduce is the emerging parallel and distributed technology to process big datasets on Hadoop Cluster. To mine big datasets it is essential to re-design the data mining algorithm on this new paradigm. In this paper, we implement three variations of Apriori algorithm using data structures hash tree, trie and hash table trie i.e. trie with hash technique on MapReduce paradigm. We emphasize and investigate the significance of these three data structures for Apriori algorithm on Hadoop cluster, which has not been given attention yet. Experiments are carried out on both real life and synthetic datasets which shows that hash table trie data structures performs far better than trie and hash tree in terms of execution time. Moreover the performance in case of hash tree becomes worst.