Curvature of high-dimensional data

TL;DR

This paper investigates the challenges of estimating curvature in high-dimensional noisy data, revealing bias issues and proposing a probabilistic framework for more accurate curvature estimation, validated on high-dimensional spheres.

Contribution

It introduces a probabilistic approach to improve curvature estimation in high-dimensional noisy datasets, addressing bias limitations of existing methods.

Findings

Bias in curvature estimates increases with dimension.

Naive estimators often fail to approximate true curvature in high dimensions.

Proposed framework improves estimation accuracy on high-dimensional spheres.

Abstract

We consider the problem of estimating curvature where the data can be viewed as a noisy sample from an underlying manifold. For manifolds of dimension greater than one there are multiple definitions of local curvature, each suggesting a different estimation process for a given data set. Recently, there has been progress in proving that estimates of ``local point cloud curvature" converge to the related smooth notion of local curvature as the density of the point cloud approaches infinity. Herein we investigate practical limitations of such convergence theorems and discuss the significant impact of bias in such estimates as reported in recent literature. We provide theoretical arguments for the fact that bias increases drastically in higher dimensions, so much so that in high dimensions, the probability that a naive curvature estimate lies in a small interval near the true curvature could be near zero. We present a probabilistic framework that enables the construction of more accurate estimators of curvature for arbitrary noise models. The efficacy of our technique is supported with experiments on spheres of dimension as large as twelve.