Revisiting Structured Variational Autoencoders

TL;DR

This paper demonstrates that modern implementations of structured variational autoencoders (SVAEs) improve accuracy and efficiency, especially for sequential data, by leveraging structure, hardware acceleration, and self-supervised training.

Contribution

The paper introduces a modern, hardware-accelerated implementation of SVAEs, showing their advantages over general models in accuracy, efficiency, and handling missing data.

Findings

SVAEs outperform general models in prediction accuracy.

Structured priors improve posterior estimates.

Self-supervised training with missing data is effective.

Abstract

Structured variational autoencoders (SVAEs) combine probabilistic graphical model priors on latent variables, deep neural networks to link latent variables to observed data, and structure-exploiting algorithms for approximate posterior inference. These models are particularly appealing for sequential data, where the prior can capture temporal dependencies. However, despite their conceptual elegance, SVAEs have proven difficult to implement, and more general approaches have been favored in practice. Here, we revisit SVAEs using modern machine learning tools and demonstrate their advantages over more general alternatives in terms of both accuracy and efficiency. First, we develop a modern implementation for hardware acceleration, parallelization, and automatic differentiation of the message passing algorithms at the core of the SVAE. Second, we show that by exploiting structure in the prior, the SVAE learns more accurate models and posterior distributions, which translate into improved performance on prediction tasks. Third, we show how the SVAE can naturally handle missing data, and we leverage this ability to develop a novel, self-supervised training approach. Altogether, these results show that the time is ripe to revisit structured variational autoencoders.