Enabling DBSCAN for Very Large-Scale High-Dimensional Spaces

TL;DR

This paper introduces a spectral data compression-based adaptation of DBSCAN, significantly improving its efficiency and accuracy for clustering large-scale, high-dimensional datasets by reducing redundancy and noise.

Contribution

We propose a novel spectral data compression technique for DBSCAN that enhances its scalability and effectiveness in high-dimensional, large-scale data analysis.

Findings

Enhanced clustering accuracy on large datasets

Reduced computational complexity of DBSCAN

Effective noise and redundancy removal

Abstract

DBSCAN is one of the most important non-parametric unsupervised data analysis tools. By applying DBSCAN to a dataset, two key analytical results can be obtained: (1) clustering data points based on density distribution and (2) identifying outliers in the dataset. However, the time complexity of the DBSCAN algorithm is $O(n^2 \beta)$, where $n$ is the number of data points and $\beta = O(D)$, with $D$ representing the dimensionality of the data space. As a result, DBSCAN becomes computationally infeasible when both $n$ and $D$ are large. In this paper, we propose a DBSCAN method based on spectral data compression, capable of efficiently processing datasets with a large number of data points ($n$) and high dimensionality ($D$). By preserving only the most critical structural information during the compression process, our method effectively removes substantial redundancy and noise. Consequently, the solution quality of DBSCAN is significantly improved, enabling more accurate and reliable results.