How do Variational Autoencoders Learn? Insights from Representational Similarity

TL;DR

This paper investigates how Variational Autoencoders learn internal representations, revealing that encoders develop stable representations early and are consistent across different hyperparameters and datasets, using representational similarity measures.

Contribution

It provides the first layerwise comparison of VAE representations, showing encoder stability and independence from hyperparameters and datasets.

Findings

Encoders' representations are learned before decoders'

Representations in encoder layers are consistent across hyperparameters

Encoder representations are stable across datasets

Abstract

The ability of Variational Autoencoders (VAEs) to learn disentangled representations has made them popular for practical applications. However, their behaviour is not yet fully understood. For example, the questions of when they can provide disentangled representations, or suffer from posterior collapse are still areas of active research. Despite this, there are no layerwise comparisons of the representations learned by VAEs, which would further our understanding of these models. In this paper, we thus look into the internal behaviour of VAEs using representational similarity techniques. Specifically, using the CKA and Procrustes similarities, we found that the encoders' representations are learned long before the decoders', and this behaviour is independent of hyperparameters, learning objectives, and datasets. Moreover, the encoders' representations in all but the mean and variance layers are similar across hyperparameters and learning objectives.