Discrete Denoising Flows

TL;DR

This paper introduces Discrete Denoising Flows, a new invertible model for categorical data that improves lossless compression and can be trained locally without gradient bias, outperforming previous models on several benchmarks.

Contribution

The paper presents Discrete Denoising Flows, a novel discrete flow model that enables local training without gradient bias and achieves better likelihood scores.

Findings

DDFs outperform Discrete Flows on toy, MNIST, and Cityscapes datasets.

DDFs enable lossless compression without data dequantization.

The model is trained locally, avoiding gradient bias.

Abstract

Discrete flow-based models are a recently proposed class of generative models that learn invertible transformations for discrete random variables. Since they do not require data dequantization and maximize an exact likelihood objective, they can be used in a straight-forward manner for lossless compression. In this paper, we introduce a new discrete flow-based model for categorical random variables: Discrete Denoising Flows (DDFs). In contrast with other discrete flow-based models, our model can be locally trained without introducing gradient bias. We show that DDFs outperform Discrete Flows on modeling a toy example, binary MNIST and Cityscapes segmentation maps, measured in log-likelihood.