Distributed Clustering based on Distributional Kernel

TL;DR

This paper presents a novel distributed clustering framework using distributional kernels that guarantees equivalence to centralized results, reduces runtime, and handles arbitrary cluster shapes, outperforming existing methods.

Contribution

Introduces the first distributed clustering framework based on distributional kernels, ensuring equivalence to centralized clustering and improved efficiency.

Findings

KDC guarantees equivalent clustering as centralized methods.

KDC reduces maximum runtime cost at distributed sites.

Kernel Bounded Cluster Cores outperforms existing algorithms.

Abstract

This paper introduces a new framework for clustering in a distributed network called Distributed Clustering based on Distributional Kernel (K) or KDC that produces the final clusters based on the similarity with respect to the distributions of initial clusters, as measured by K. It is the only framework that satisfies all three of the following properties. First, KDC guarantees that the combined clustering outcome from all sites is equivalent to the clustering outcome of its centralized counterpart from the combined dataset from all sites. Second, the maximum runtime cost of any site in distributed mode is smaller than the runtime cost in centralized mode. Third, it is designed to discover clusters of arbitrary shapes, sizes and densities. To the best of our knowledge, this is the first distributed clustering framework that employs a distributional kernel. The distribution-based clustering leads directly to significantly better clustering outcomes than existing methods of distributed clustering. In addition, we introduce a new clustering algorithm called Kernel Bounded Cluster Cores, which is the best clustering algorithm applied to KDC among existing clustering algorithms. We also show that KDC is a generic framework that enables a quadratic time clustering algorithm to deal with large datasets that would otherwise be impossible.