Be More Active! Understanding the Differences between Mean and Sampled Representations of Variational Autoencoders

TL;DR

This paper investigates the differences between mean and sampled representations in Variational Autoencoders, revealing that passive variables cause higher correlation in mean representations but do not affect disentanglement, informing better downstream application strategies.

Contribution

It extends the concept of selective posterior collapse to multiple data examples and isolates passive variables responsible for correlation discrepancies in mean versus sampled representations.

Findings

Passive variables cause higher correlation in mean representations.

Mean and sampled active variables are equally disentangled.

Removing passive variables can improve downstream task performance.

Abstract

The ability of Variational Autoencoders to learn disentangled representations has made them appealing for practical applications. However, their mean representations, which are generally used for downstream tasks, have recently been shown to be more correlated than their sampled counterpart, on which disentanglement is usually measured. In this paper, we refine this observation through the lens of selective posterior collapse, which states that only a subset of the learned representations, the active variables, is encoding useful information while the rest (the passive variables) is discarded. We first extend the existing definition to multiple data examples and show that active variables are equally disentangled in mean and sampled representations. Based on this extension and the pre-trained models from disentanglement lib, we then isolate the passive variables and show that they are responsible for the discrepancies between mean and sampled representations. Specifically, passive variables exhibit high correlation scores with other variables in mean representations while being fully uncorrelated in sampled ones. We thus conclude that despite what their higher correlation might suggest, mean representations are still good candidates for downstream tasks applications. However, it may be beneficial to remove their passive variables, especially when used with models sensitive to correlated features.