CNAK : Cluster Number Assisted K-means

TL;DR

This paper introduces CNAK, a Monte-Carlo based method for determining the number of clusters in large, high-dimensional datasets by analyzing cluster stability, which also helps identify cluster representatives and hierarchy.

Contribution

The paper presents a novel Monte-Carlo simulation approach for robustly predicting cluster numbers and representatives, improving speed and accuracy over existing methods.

Findings

Effective in identifying a single cluster

Handles high-dimensional and imbalanced datasets well

Capable of indicating cluster hierarchy

Abstract

Determining the number of clusters present in a dataset is an important problem in cluster analysis. Conventional clustering techniques generally assume this parameter to be provided up front. %user supplied. %Recently, robustness of any given clustering algorithm is analyzed to measure cluster stability/instability which in turn determines the cluster number. In this paper, we propose a method which analyzes cluster stability for predicting the cluster number. Under the same computational framework, the technique also finds representatives of the clusters. The method is apt for handling big data, as we design the algorithm using \emph{Monte-Carlo} simulation. Also, we explore a few pertinent issues found to be of also clustering. Experiments reveal that the proposed method is capable of identifying a single cluster. It is robust in handling high dimensional dataset and performs reasonably well over datasets having cluster imbalance. Moreover, it can indicate cluster hierarchy, if present. Overall we have observed significant improvement in speed and quality for predicting cluster numbers as well as the composition of clusters in a large dataset.