Grid-based Approaches for Distributed Data Mining Applications

TL;DR

This paper introduces grid-based methods for distributed clustering and frequent itemset generation, evaluating their performance on an experimental grid system and analyzing the challenges in achieving realistic performance expectations.

Contribution

It presents new distributed data mining algorithms tailored for grid environments and provides a performance evaluation with comparison to an analytical model.

Findings

Distributed algorithms are well-adapted for grid environments

Performance on grid systems shows significant overheads

Realistic performance expectations are challenging to achieve

Abstract

The data mining field is an important source of large-scale applications and datasets which are getting more and more common. In this paper, we present grid-based approaches for two basic data mining applications, and a performance evaluation on an experimental grid environment that provides interesting monitoring capabilities and configuration tools. We propose a new distributed clustering approach and a distributed frequent itemsets generation well-adapted for grid environments. Performance evaluation is done using the Condor system and its workflow manager DAGMan. We also compare this performance analysis to a simple analytical model to evaluate the overheads related to the workflow engine and the underlying grid system. This will specifically show that realistic performance expectations are currently difficult to achieve on the grid.