Scalable Density-Based Distributed Clustering

TL;DR

This paper introduces a scalable distributed clustering algorithm that balances clustering quality and communication cost by selecting local representatives for global density-based clustering, suitable for large heterogeneous datasets.

Contribution

It proposes a novel scalable density-based distributed clustering method that efficiently selects local representatives to improve global clustering performance and reduce data transmission.

Findings

High-quality clusterings achieved with scalable transmission costs

Efficient local representative selection process

Effective global clustering on large heterogeneous data

Abstract

Clustering has become an increasingly important task in analysing huge amounts of data. Traditional applications require that all data has to be located at the site where it is scrutinized. Nowadays, large amounts of heterogeneous, complex data reside on different, independently working computers which are connected to each other via local or wide area networks. In this paper, we propose a scalable density-based distributed clustering algorithm which allows a user-defined trade-off between clustering quality and the number of transmitted objects from the different local sites to a global server site. Our approach consists of the following steps: First, we order all objects located at a local site according to a quality criterion reflecting their suitability to serve as local representatives. Then we send the best of these representatives to a server site where they are clustered with a slightly enhanced density-based clustering algorithm. This approach is very efficient, because the local detemination of suitable representatives can be carried out quickly and independently from each other. Furthermore, based on the scalable number of the most suitable local representatives, the global clustering can be done very effectively and efficiently. In our experimental evaluation, we will show that our new scalable density-based distributed clustering approach results in high quality clusterings with scalable transmission cost.