Diffusion bridges vector quantized Variational AutoEncoders

TL;DR

This paper introduces a diffusion bridge approach for VQ-VAE models, enabling simultaneous training of the prior and encoder/decoder, resulting in more efficient and end-to-end generative modeling.

Contribution

It proposes a novel diffusion bridge method that integrates prior training with VQ-VAE, simplifying the generative process and improving efficiency.

Findings

Competitive with autoregressive priors on mini-ImageNet and CIFAR datasets

Enables end-to-end training of VQ-VAE models

Improves efficiency in optimization and sampling processes

Abstract

Vector Quantized-Variational AutoEncoders (VQ-VAE) are generative models based on discrete latent representations of the data, where inputs are mapped to a finite set of learned embeddings.To generate new samples, an autoregressive prior distribution over the discrete states must be trained separately. This prior is generally very complex and leads to slow generation. In this work, we propose a new model to train the prior and the encoder/decoder networks simultaneously. We build a diffusion bridge between a continuous coded vector and a non-informative prior distribution. The latent discrete states are then given as random functions of these continuous vectors. We show that our model is competitive with the autoregressive prior on the mini-Imagenet and CIFAR dataset and is efficient in both optimization and sampling. Our framework also extends the standard VQ-VAE and enables end-to-end training.