Data Partitioning View of Mining Big Data

TL;DR

This paper discusses data partitioning as a key strategy for mining big data, highlighting how dividing data into subsets enables in-memory analysis and the synthesis of global patterns from local ones.

Contribution

It revisits and emphasizes the importance of data partitioning in big data mining, presenting findings on local pattern discovery and their role in understanding global data.

Findings

Partitioning enables in-memory analysis of large datasets.

Local patterns can be synthesized to identify global patterns.

Partitioning strategies significantly impact mining effectiveness.

Abstract

There are two main approximations of mining big data in memory. One is to partition a big dataset to several subsets, so as to mine each subset in memory. By this way, global patterns can be obtained by synthesizing all local patterns discovered from these subsets. Another is the statistical sampling method. This indicates that data partitioning should be an important strategy for mining big data. This paper recalls our work on mining big data with a data partitioning and shows some interesting findings among the local patterns discovered from subsets of a dataset.