The Local Dimension of Deep Manifold

TL;DR

This paper introduces a SVD-based method to estimate the local intrinsic dimension of deep manifolds in CNNs, revealing how dimensions decline across layers and vary with different categories, providing insights into neural network structure.

Contribution

It proposes a novel SVD-based approach to estimate the local dimension of deep manifolds in CNNs and analyzes how these dimensions change across layers and categories.

Findings

Dimensions decline rapidly along layers

Gaussian noise approximates intrinsic dimension

Different categories have similar manifold dimensions

Abstract

Based on our observation that there exists a dramatic drop for the singular values of the fully connected layers or a single feature map of the convolutional layer, and that the dimension of the concatenated feature vector almost equals the summation of the dimension on each feature map, we propose a singular value decomposition (SVD) based approach to estimate the dimension of the deep manifolds for a typical convolutional neural network VGG19. We choose three categories from the ImageNet, namely Persian Cat, Container Ship and Volcano, and determine the local dimension of the deep manifolds of the deep layers through the tangent space of a target image. Through several augmentation methods, we found that the Gaussian noise method is closer to the intrinsic dimension, as by adding random noise to an image we are moving in an arbitrary dimension, and when the rank of the feature matrix of the augmented images does not increase we are very close to the local dimension of the manifold. We also estimate the dimension of the deep manifold based on the tangent space for each of the maxpooling layers. Our results show that the dimensions of different categories are close to each other and decline quickly along the convolutional layers and fully connected layers. Furthermore, we show that the dimensions decline quickly inside the Conv5 layer. Our work provides new insights for the intrinsic structure of deep neural networks and helps unveiling the inner organization of the black box of deep neural networks.