Transportation analysis of denoising autoencoders: a novel method for analyzing deep neural networks

TL;DR

This paper introduces a transportation-based framework to analyze deep neural networks, specifically denoising autoencoders, revealing how they transport data distributions and relate to Wasserstein gradient flows.

Contribution

It formulates feature maps as transport maps, providing a novel analytical approach to understanding deep neural network behavior through transportation dynamics.

Findings

The transport map of the DAE is derived.

Deep DAEs transport data mass to reduce entropy.

Connections to Wasserstein gradient flows are established.

Abstract

The feature map obtained from the denoising autoencoder (DAE) is investigated by determining transportation dynamics of the DAE, which is a cornerstone for deep learning. Despite the rapid development in its application, deep neural networks remain analytically unexplained, because the feature maps are nested and parameters are not faithful. In this paper, we address the problem of the formulation of nested complex of parameters by regarding the feature map as a transport map. Even when a feature map has different dimensions between input and output, we can regard it as a transportation map by considering that both the input and output spaces are embedded in a common high-dimensional space. In addition, the trajectory is a geometric object and thus, is independent of parameterization. In this manner, transportation can be regarded as a universal character of deep neural networks. By determining and analyzing the transportation dynamics, we can understand the behavior of a deep neural network. In this paper, we investigate a fundamental case of deep neural networks: the DAE. We derive the transport map of the DAE, and reveal that the infinitely deep DAE transports mass to decrease a certain quantity, such as entropy, of the data distribution. These results though analytically simple, shed light on the correspondence between deep neural networks and the Wasserstein gradient flows.