The Curse Revisited: When are Distances Informative for the Ground Truth in Noisy High-Dimensional Data?

TL;DR

This paper analyzes how noise affects the informativeness of distances in high-dimensional data, showing that neighborhood relations can remain meaningful under certain conditions despite noise and concentration effects.

Contribution

It provides an exact probabilistic characterization of noisy distances in high dimensions and identifies conditions where neighborhood relations remain truthful despite noise.

Findings

Distances can still be informative under noise when decomposed into ground truth and noise components.

The derived phase shift correlates with the performance of dimensionality reduction methods.

Empirical results validate the theoretical phase shift and its impact on data reconstruction.

Abstract

Distances between data points are widely used in machine learning applications. Yet, when corrupted by noise, these distances -- and thus the models based upon them -- may lose their usefulness in high dimensions. Indeed, the small marginal effects of the noise may then accumulate quickly, shifting empirical closest and furthest neighbors away from the ground truth. In this paper, we exactly characterize such effects in noisy high-dimensional data using an asymptotic probabilistic expression. Previously, it has been argued that neighborhood queries become meaningless and unstable when distance concentration occurs, which means that there is a poor relative discrimination between the furthest and closest neighbors in the data. However, we conclude that this is not necessarily the case when we decompose the data in a ground truth -- which we aim to recover -- and noise component. More specifically, we derive that under particular conditions, empirical neighborhood relations affected by noise are still likely to be truthful even when distance concentration occurs. We also include thorough empirical verification of our results, as well as interesting experiments in which our derived 'phase shift' where neighbors become random or not turns out to be identical to the phase shift where common dimensionality reduction methods perform poorly or well for recovering low-dimensional reconstructions of high-dimensional data with dense noise.