On a Mechanism Framework of Autoencoders

TL;DR

This paper develops a theoretical framework for understanding autoencoders, focusing on their properties, generalization mechanisms, and explanations for various autoencoder types and neural network models.

Contribution

It introduces a formal framework for autoencoder mechanisms, explaining their properties, generalization, and applications to different neural network architectures.

Findings

Autoencoders satisfy bijective and data disentangling properties.

The framework explains variational, denoising, and linear autoencoders.

Autoencoders outperform PCA and decision trees in specific tasks.

Abstract

This paper proposes a theoretical framework on the mechanism of autoencoders. To the encoder part, under the main use of dimensionality reduction, we investigate its two fundamental properties: bijective maps and data disentangling. The general construction methods of an encoder that satisfies either or both of the above two properties are given. The generalization mechanism of autoencoders is modeled. Based on the theoretical framework above, we explain some experimental results of variational autoencoders, denoising autoencoders, and linear-unit autoencoders, with emphasis on the interpretation of the lower-dimensional representation of data via encoders; and the mechanism of image restoration through autoencoders is natural to be understood by those explanations. Compared to PCA and decision trees, the advantages of (generalized) autoencoders on dimensionality reduction and classification are demonstrated, respectively. Convolutional neural networks and randomly weighted neural networks are also interpreted by this framework.