ABID: Angle Based Intrinsic Dimensionality

TL;DR

This paper introduces a novel angle-based method for estimating local intrinsic dimensionality of data, providing an alternative to distance-based approaches and enhancing understanding of data complexity.

Contribution

The paper presents a new intrinsic dimensionality estimator based on angle distributions, independent of distance measures, and demonstrates its effectiveness alongside existing methods.

Findings

Angle-based estimator aligns with distance-based measures

Provides complementary insights into data complexity

Enhances understanding of local intrinsic dimensionality

Abstract

The intrinsic dimensionality refers to the ``true'' dimensionality of the data, as opposed to the dimensionality of the data representation. For example, when attributes are highly correlated, the intrinsic dimensionality can be much lower than the number of variables. Local intrinsic dimensionality refers to the observation that this property can vary for different parts of the data set; and intrinsic dimensionality can serve as a proxy for the local difficulty of the data set. Most popular methods for estimating the local intrinsic dimensionality are based on distances, and the rate at which the distances to the nearest neighbors increase, a concept known as ``expansion dimension''. In this paper we introduce an orthogonal concept, which does not use any distances: we use the distribution of angles between neighbor points. We derive the theoretical distribution of angles and use this to construct an estimator for intrinsic dimensionality. Experimentally, we verify that this measure behaves similarly, but complementarily, to existing measures of intrinsic dimensionality. By introducing a new idea of intrinsic dimensionality to the research community, we hope to contribute to a better understanding of intrinsic dimensionality and to spur new research in this direction.