Improved Variational Inference in Discrete VAEs using Error Correcting Codes

TL;DR

This paper introduces a novel approach to improve discrete Variational Autoencoders by integrating Error-Correcting Codes, enhancing inference accuracy, generation quality, and uncertainty calibration through a communication system perspective.

Contribution

It proposes using Error-Correcting Codes in discrete VAEs to reduce the variational gap and improve inference, a novel perspective in deep probabilistic modeling.

Findings

Significant improvements in generation quality and data reconstruction.

Enhanced uncertainty calibration in discrete VAEs.

Outperforms models trained with tighter bounds like IWAE.

Abstract

Despite advances in deep probabilistic models, learning discrete latent representations remains challenging. This work introduces a novel method to improve inference in discrete Variational Autoencoders by reframing the inference problem through a generative perspective. We conceptualize the model as a communication system, and propose to leverage Error-Correcting Codes (ECCs) to introduce redundancy in latent representations, allowing the variational posterior to produce more accurate estimates and reduce the variational gap. We present a proof-of-concept using a Discrete Variational Autoencoder with binary latent variables and low-complexity repetition codes, extending it to a hierarchical structure for disentangling global and local data features. Our approach significantly improves generation quality, data reconstruction, and uncertainty calibration, outperforming the uncoded models even when trained with tighter bounds such as the Importance Weighted Autoencoder objective. We also outline the properties that ECCs should possess to be effectively utilized for improved discrete variational inference.