Trading Information between Latents in Hierarchical Variational Autoencoders

TL;DR

This paper explores how to control and optimize information flow in hierarchical variational autoencoders by splitting the rate across layers, providing theoretical bounds and practical guidance for various applications.

Contribution

It introduces a method to decompose the information rate in hierarchical VAEs, enabling independent tuning of each layer's contribution and offering theoretical bounds on downstream task performance.

Findings

Layer-wise rate splitting is feasible in hierarchical VAEs.

Theoretical bounds relate layer rates to task performance.

Guidelines for rate-space targeting in applications.

Abstract

Variational Autoencoders (VAEs) were originally motivated (Kingma & Welling, 2014) as probabilistic generative models in which one performs approximate Bayesian inference. The proposal of $\beta$-VAEs (Higgins et al., 2017) breaks this interpretation and generalizes VAEs to application domains beyond generative modeling (e.g., representation learning, clustering, or lossy data compression) by introducing an objective function that allows practitioners to trade off between the information content ("bit rate") of the latent representation and the distortion of reconstructed data (Alemi et al., 2018). In this paper, we reconsider this rate/distortion trade-off in the context of hierarchical VAEs, i.e., VAEs with more than one layer of latent variables. We identify a general class of inference models for which one can split the rate into contributions from each layer, which can then be tuned independently. We derive theoretical bounds on the performance of downstream tasks as functions of the individual layers' rates and verify our theoretical findings in large-scale experiments. Our results provide guidance for practitioners on which region in rate-space to target for a given application.