Verifying the Union of Manifolds Hypothesis for Image Data

TL;DR

This paper challenges the traditional manifold hypothesis for image data, proposing and empirically verifying that data lies on a union of manifolds with varying dimensions, which improves deep learning models' performance.

Contribution

It introduces the union of manifolds hypothesis for image data, providing empirical evidence and demonstrating its benefits for deep learning tasks.

Findings

Data lies on a disconnected set with varying intrinsic dimensions.

Models with inductive bias for union of manifolds outperform traditional approaches.

Improved performance in classification and generative modeling tasks.

Abstract

Deep learning has had tremendous success at learning low-dimensional representations of high-dimensional data. This success would be impossible if there was no hidden low-dimensional structure in data of interest; this existence is posited by the manifold hypothesis, which states that the data lies on an unknown manifold of low intrinsic dimension. In this paper, we argue that this hypothesis does not properly capture the low-dimensional structure typically present in image data. Assuming that data lies on a single manifold implies intrinsic dimension is identical across the entire data space, and does not allow for subregions of this space to have a different number of factors of variation. To address this deficiency, we consider the union of manifolds hypothesis, which states that data lies on a disjoint union of manifolds of varying intrinsic dimensions. We empirically verify this hypothesis on commonly-used image datasets, finding that indeed, observed data lies on a disconnected set and that intrinsic dimension is not constant. We also provide insights into the implications of the union of manifolds hypothesis in deep learning, both supervised and unsupervised, showing that designing models with an inductive bias for this structure improves performance across classification and generative modelling tasks. Our code is available at https://github.com/layer6ai-labs/UoMH.