PixelVAE: A Latent Variable Model for Natural Images

TL;DR

PixelVAE combines the strengths of VAEs and PixelCNN to effectively model natural images, capturing both global structure and fine details with fewer autoregressive layers and hierarchical latent variables.

Contribution

It introduces PixelVAE, a novel model integrating VAE and PixelCNN, with hierarchical latent variables, achieving state-of-the-art image modeling performance.

Findings

State-of-the-art on binarized MNIST

Competitive on 64x64 ImageNet

High-quality samples on LSUN bedrooms

Abstract

Natural image modeling is a landmark challenge of unsupervised learning. Variational Autoencoders (VAEs) learn a useful latent representation and model global structure well but have difficulty capturing small details. PixelCNN models details very well, but lacks a latent code and is difficult to scale for capturing large structures. We present PixelVAE, a VAE model with an autoregressive decoder based on PixelCNN. Our model requires very few expensive autoregressive layers compared to PixelCNN and learns latent codes that are more compressed than a standard VAE while still capturing most non-trivial structure. Finally, we extend our model to a hierarchy of latent variables at different scales. Our model achieves state-of-the-art performance on binarized MNIST, competitive performance on 64x64 ImageNet, and high-quality samples on the LSUN bedrooms dataset.