On perfect clustering of high dimension, low sample size data

TL;DR

This paper introduces MADD, a dissimilarity measure that leverages the distance concentration phenomenon to improve clustering performance in high-dimensional, low-sample-size data, and proposes a new estimator for the number of clusters.

Contribution

It presents MADD as an alternative to Euclidean distance for HDLSS data and develops a consistent estimator for the number of clusters based on penalized Dunn index.

Findings

MADD improves clustering accuracy in HDLSS scenarios

Existing cluster number estimation algorithms perform better with MADD

The proposed estimator is consistent in HDLSS asymptotics

Abstract

Popular clustering algorithms based on usual distance functions (e.g., Euclidean distance) often suffer in high dimension, low sample size (HDLSS) situations, where concentration of pairwise distances has adverse effects on their performance. In this article, we use a dissimilarity measure based on the data cloud, called MADD, which takes care of this problem. MADD uses the distance concentration phenomenon to its advantage, and as a result, clustering algorithms based on MADD usually perform better for high dimensional data. Using theoretical and numerical results, we amply demonstrate it in this article. We also address the problem of estimating the number of clusters. This is a very challenging problem in cluster analysis, and several algorithms have been proposed for it. We show that many of these existing algorithms have superior performance in high dimensions when MADD is used instead of the Euclidean distance. We also construct a new estimator based on penalized Dunn index and prove its consistency in the HDLSS asymptotic regime, where the sample size remains fixed and the dimension grows to infinity. Several simulated and real data sets are analyzed to demonstrate the importance of MADD for cluster analysis of high dimensional data.