Embrace the Gap: VAEs Perform Independent Mechanism Analysis

TL;DR

This paper explains why variational autoencoders (VAEs) effectively learn representations by demonstrating their connection to independent mechanism analysis (IMA) and showing they can recover true latent factors under certain conditions.

Contribution

It proves that in the near-deterministic decoder regime, VAEs' optimal encoder approximately inverts the decoder, linking ELBO maximization to IMA and improving understanding of their success.

Findings

VAEs perform independent mechanism analysis (IMA) under certain conditions.

The ELBO converges to a regularized log-likelihood, aiding representation learning.

VAEs recover true latent factors in synthetic and image data when IMA assumptions hold.

Abstract

Variational autoencoders (VAEs) are a popular framework for modeling complex data distributions; they can be efficiently trained via variational inference by maximizing the evidence lower bound (ELBO), at the expense of a gap to the exact (log-)marginal likelihood. While VAEs are commonly used for representation learning, it is unclear why ELBO maximization would yield useful representations, since unregularized maximum likelihood estimation cannot invert the data-generating process. Yet, VAEs often succeed at this task. We seek to elucidate this apparent paradox by studying nonlinear VAEs in the limit of near-deterministic decoders. We first prove that, in this regime, the optimal encoder approximately inverts the decoder -- a commonly used but unproven conjecture -- which we refer to as {\em self-consistency}. Leveraging self-consistency, we show that the ELBO converges to a regularized log-likelihood. This allows VAEs to perform what has recently been termed independent mechanism analysis (IMA): it adds an inductive bias towards decoders with column-orthogonal Jacobians, which helps recovering the true latent factors. The gap between ELBO and log-likelihood is therefore welcome, since it bears unanticipated benefits for nonlinear representation learning. In experiments on synthetic and image data, we show that VAEs uncover the true latent factors when the data generating process satisfies the IMA assumption.