Dimension Estimation Using Random Connection Models

TL;DR

This paper introduces a new intrinsic dimension estimator based on binary neighborhood graphs modeled by a random connection model, which is computationally efficient and does not require explicit distance data.

Contribution

The paper proposes a novel intrinsic dimension estimator using adjacency matrices and a random connection model, with proven asymptotic properties and competitive performance.

Findings

Estimator scales as n log n in computation

Asymptotic distribution and convergence rate are specified

Performs favorably compared to existing methods

Abstract

Information about intrinsic dimension is crucial to perform dimensionality reduction, compress information, design efficient algorithms, and do statistical adaptation. In this paper we propose an estimator for the intrinsic dimension of a data set. The estimator is based on binary neighbourhood information about the observations in the form of two adjacency matrices, and does not require any explicit distance information. The underlying graph is modelled according to a subset of a specific random connection model, sometimes referred to as the Poisson blob model. Computationally the estimator scales like n log n, and we specify its asymptotic distribution and rate of convergence. A simulation study on both real and simulated data shows that our approach compares favourably with some competing methods from the literature, including approaches that rely on distance information.