Merging $K$-means with hierarchical clustering for identifying general-shaped groups

TL;DR

This paper introduces a hybrid clustering method combining K-means and hierarchical clustering to effectively identify arbitrarily shaped groups in large datasets, overcoming limitations of each individual approach.

Contribution

A novel non-parametric hybrid clustering approach that merges K-means and hierarchical methods for detecting general-shaped clusters in large datasets.

Findings

Effective in identifying complex-shaped clusters

Performs well on simulated datasets

Scalable to larger datasets

Abstract

Clustering partitions a dataset such that observations placed together in a group are similar but different from those in other groups. Hierarchical and $K$-means clustering are two approaches but have different strengths and weaknesses. For instance, hierarchical clustering identifies groups in a tree-like structure but suffers from computational complexity in large datasets while $K$-means clustering is efficient but designed to identify homogeneous spherically-shaped clusters. We present a hybrid non-parametric clustering approach that amalgamates the two methods to identify general-shaped clusters and that can be applied to larger datasets. Specifically, we first partition the dataset into spherical groups using $K$-means. We next merge these groups using hierarchical methods with a data-driven distance measure as a stopping criterion. Our proposal has the potential to reveal groups with general shapes and structure in a dataset. We demonstrate good performance on several simulated and real datasets.