From Principal Subspaces to Principal Components with Linear Autoencoders

TL;DR

This paper demonstrates how to recover principal component loading vectors directly from the weights of a linear autoencoder, bridging the gap between autoencoder training and principal component analysis.

Contribution

It introduces a method to extract principal component loading vectors from the weights of a linear autoencoder, clarifying their relationship.

Findings

Autoencoder weights span the same subspace as principal components

A method to recover principal component vectors from autoencoder weights

Provides theoretical insight into autoencoder and PCA connection

Abstract

The autoencoder is an effective unsupervised learning model which is widely used in deep learning. It is well known that an autoencoder with a single fully-connected hidden layer, a linear activation function and a squared error cost function trains weights that span the same subspace as the one spanned by the principal component loading vectors, but that they are not identical to the loading vectors. In this paper, we show how to recover the loading vectors from the autoencoder weights.