The Intrinsic Dimension of Images and Its Impact on Learning

TL;DR

This paper investigates the low-dimensional structure of natural images and demonstrates its significance in deep learning, showing that datasets with lower intrinsic dimension are easier to learn and generalize better.

Contribution

The authors apply and validate dimension estimation methods on real datasets, revealing the low intrinsic dimension of natural images and its impact on neural network learning and generalization.

Findings

Natural image datasets have low intrinsic dimension relative to pixel count.

Lower intrinsic dimension correlates with easier learning for neural networks.

Models trained on low-dimensional data generalize better from training to test sets.

Abstract

It is widely believed that natural image data exhibits low-dimensional structure despite the high dimensionality of conventional pixel representations. This idea underlies a common intuition for the remarkable success of deep learning in computer vision. In this work, we apply dimension estimation tools to popular datasets and investigate the role of low-dimensional structure in deep learning. We find that common natural image datasets indeed have very low intrinsic dimension relative to the high number of pixels in the images. Additionally, we find that low dimensional datasets are easier for neural networks to learn, and models solving these tasks generalize better from training to test data. Along the way, we develop a technique for validating our dimension estimation tools on synthetic data generated by GANs allowing us to actively manipulate the intrinsic dimension by controlling the image generation process. Code for our experiments may be found here https://github.com/ppope/dimensions.