A New Estimator of Intrinsic Dimension Based on the Multipoint Morisita Index

TL;DR

This paper introduces a robust, efficient estimator of intrinsic dimension using the multipoint Morisita index, capable of handling large, noisy datasets and providing accurate manifold dimension estimates.

Contribution

The paper presents a novel intrinsic dimension estimator based on the multipoint Morisita index, improving robustness and efficiency over existing methods.

Findings

Robust to sample size and noise

Unaffected by edge effects

Computationally efficient for large datasets

Abstract

The size of datasets has been increasing rapidly both in terms of number of variables and number of events. As a result, the empty space phenomenon and the curse of dimensionality complicate the extraction of useful information. But, in general, data lie on non-linear manifolds of much lower dimension than that of the spaces in which they are embedded. In many pattern recognition tasks, learning these manifolds is a key issue and it requires the knowledge of their true intrinsic dimension. This paper introduces a new estimator of intrinsic dimension based on the multipoint Morisita index. It is applied to both synthetic and real datasets of varying complexities and comparisons with other existing estimators are carried out. The proposed estimator turns out to be fairly robust to sample size and noise, unaffected by edge effects, able to handle large datasets and computationally efficient.