A Matrix Logic Approach to Efficient Frequent Itemset Discovery in Large Data Sets

TL;DR

This paper introduces a Boolean matrix-based frequent itemset mining algorithm that improves efficiency and scalability in large datasets by leveraging matrix operations instead of candidate generation.

Contribution

The proposed algorithm uses Boolean matrix operations for frequent itemset mining, reducing storage and computational bottlenecks in high-dimensional, large-scale transaction databases.

Findings

Efficiently mines frequent itemsets with low support thresholds.

Maintains good performance as transaction numbers increase.

Demonstrates scalability and robustness in large datasets.

Abstract

This paper proposes a frequent itemset mining algorithm based on the Boolean matrix method, aiming to solve the storage and computational bottlenecks of traditional frequent pattern mining algorithms in high-dimensional and large-scale transaction databases. By representing the itemsets in the transaction database as Boolean matrices, the algorithm uses Boolean logic operations such as AND and OR to efficiently calculate the support of the itemsets, avoiding the generation and storage of a large number of candidates itemsets in traditional algorithms. The algorithm recursively mines frequent itemsets through matrix operations and can flexibly adapt to different data scales and support thresholds. In the experiment, the public Groceries dataset was selected, and the running efficiency test and frequent itemset mining effect test were designed to evaluate the algorithm's performance indicators such as running time, memory usage, and number of frequent itemsets under different transaction numbers and support thresholds. The experimental results show that the algorithm can efficiently mine a large number of frequent itemsets when the support threshold is low, and focus on strong association rules with high support when the threshold is high. In addition, the changing trends of running time and memory usage show that the Boolean matrix method can still maintain good running efficiency when the number of transactions increases significantly and has high scalability and robustness. Future research can improve memory optimization and matrix block operations, and combine distributed computing and deep learning models to further enhance the algorithm's applicability and real-time processing capabilities in ultra-large-scale data environments. The algorithm has broad application potential and development prospects in the fields of market analysis, recommendation systems, and network security.