A Probabilistic $\ell_1$ Method for Clustering High Dimensional Data

TL;DR

This paper introduces a probabilistic iterative clustering method for high-dimensional data using the $\, ext{l}_1$-metric, addressing distance unreliability and computational complexity issues in high-dimensional spaces.

Contribution

It presents a novel $\, ext{l}_1$-based clustering algorithm that is computationally efficient and performs better as data dimensionality increases.

Findings

Algorithm complexity is linear in data dimension.

Performance improves with higher data dimensionality.

Uses weighted medians for clustering in high-dimensional space.

Abstract

In general, the clustering problem is NP-hard, and global optimality cannot be established for non-trivial instances. For high-dimensional data, distance-based methods for clustering or classification face an additional difficulty, the unreliability of distances in very high-dimensional spaces. We propose a distance-based iterative method for clustering data in very high-dimensional space, using the $\ell_1$-metric that is less sensitive to high dimensionality than the Euclidean distance. For $K$ clusters in $\mathbb{R}^n$, the problem decomposes to $K$ problems coupled by probabilities, and an iteration reduces to finding $Kn$ weighted medians of points on a line. The complexity of the algorithm is linear in the dimension of the data space, and its performance was observed to improve significantly as the dimension increases.