Clustering by Deep Nearest Neighbor Descent (D-NND): A Density-based Parameter-Insensitive Clustering Method

TL;DR

The paper introduces D-NND, a hierarchical density-based clustering method that effectively learns cluster structures without relying heavily on density estimation parameters, reducing common issues like over- and under-smoothing.

Contribution

It presents a novel deep hierarchical clustering approach that is parameter-insensitive and capable of accurately capturing underlying data structures.

Findings

D-NND effectively avoids over-smoothing and ripple noise.

The method demonstrates high reliability and insensitivity to parameter choices.

It outperforms traditional density-based clustering methods in discovering true cluster structures.

Abstract

Most density-based clustering methods largely rely on how well the underlying density is estimated. However, density estimation itself is also a challenging problem, especially the determination of the kernel bandwidth. A large bandwidth could lead to the over-smoothed density estimation in which the number of density peaks could be less than the true clusters, while a small bandwidth could lead to the under-smoothed density estimation in which spurious density peaks, or called the "ripple noise", would be generated in the estimated density. In this paper, we propose a density-based hierarchical clustering method, called the Deep Nearest Neighbor Descent (D-NND), which could learn the underlying density structure layer by layer and capture the cluster structure at the same time. The over-smoothed density estimation could be largely avoided and the negative effect of the under-estimated cases could be also largely reduced. Overall, D-NND presents not only the strong capability of discovering the underlying cluster structure but also the remarkable reliability due to its insensitivity to parameters.