# Jeffrey's prior sampling of deep sigmoidal networks

## TL;DR

This paper investigates the geometric structure of data manifolds learned by deep neural networks, using Monte Carlo sampling to analyze their shape and boundary properties, and discusses implications for sampling high-dimensional manifolds.

## Contribution

It provides a detailed analysis of the learned data manifolds in deep networks and explores the challenges of sampling in high-dimensional geometric spaces.

## Key findings

- Model manifold is a slightly elongated hyperball.
- Reconstructed data mostly appears on the boundary of the manifold.
- Discusses the relation between high-dimensional geometry and model reduction.

## Abstract

Neural networks have been shown to have a remarkable ability to uncover low dimensional structure in data: the space of possible reconstructed images form a reduced model manifold in image space. We explore this idea directly by analyzing the manifold learned by Deep Belief Networks and Stacked Denoising Autoencoders using Monte Carlo sampling. The model manifold forms an only slightly elongated hyperball with actual reconstructed data appearing predominantly on the boundaries of the manifold. In connection with the results we present, we discuss problems of sampling high-dimensional manifolds as well as recent work [M. Transtrum, G. Hart, and P. Qiu, Submitted (2014)] discussing the relation between high dimensional geometry and model reduction.