Federated Hierarchical Clustering with Automatic Selection of Optimal Cluster Numbers

TL;DR

This paper introduces Fed-$k^*$-HC, a federated clustering framework that automatically determines the optimal number of clusters by hierarchical merging of client-generated micro-subclusters, addressing unknown and imbalanced cluster scenarios.

Contribution

It proposes a novel federated clustering method that automatically identifies the optimal cluster count using hierarchical merging and micro-subclusters, improving robustness and accuracy.

Findings

Accurately determines the number of clusters in diverse datasets.

Handles varying cluster sizes and shapes effectively.

Demonstrates superior performance over existing methods.

Abstract

Federated Clustering (FC) is an emerging and promising solution in exploring data distribution patterns from distributed and privacy-protected data in an unsupervised manner. Existing FC methods implicitly rely on the assumption that clients are with a known number of uniformly sized clusters. However, the true number of clusters is typically unknown, and cluster sizes are naturally imbalanced in real scenarios. Furthermore, the privacy-preserving transmission constraints in federated learning inevitably reduce usable information, making the development of robust and accurate FC extremely challenging. Accordingly, we propose a novel FC framework named Fed-$k^*$-HC, which can automatically determine an optimal number of clusters $k^*$ based on the data distribution explored through hierarchical clustering. To obtain the global data distribution for $k^*$ determination, we let each client generate micro-subclusters. Their prototypes are then uploaded to the server for hierarchical merging. The density-based merging design allows exploring clusters of varying sizes and shapes, and the progressive merging process can self-terminate according to the neighboring relationships among the prototypes to determine $k^*$. Extensive experiments on diverse datasets demonstrate the FC capability of the proposed Fed-$k^*$-HC in accurately exploring a proper number of clusters.