Analog Bits: Generating Discrete Data using Diffusion Models with Self-Conditioning

TL;DR

This paper introduces Bit Diffusion, a diffusion model that generates discrete data by representing it as binary bits and modeling these as continuous variables, leading to improved performance in image generation and captioning.

Contribution

The paper proposes a novel approach called Bit Diffusion that models discrete data as analog bits with continuous diffusion models, enhancing sample quality and efficiency.

Findings

Outperforms previous state-of-the-art in discrete image generation on CIFAR-10 and ImageNet-64x64.

Achieves competitive results in image captioning on MS-COCO.

Introduces Self-Conditioning and Asymmetric Time Intervals techniques for improved sample quality.

Abstract

We present Bit Diffusion: a simple and generic approach for generating discrete data with continuous state and continuous time diffusion models. The main idea behind our approach is to first represent the discrete data as binary bits, and then train a continuous diffusion model to model these bits as real numbers which we call analog bits. To generate samples, the model first generates the analog bits, which are then thresholded to obtain the bits that represent the discrete variables. We further propose two simple techniques, namely Self-Conditioning and Asymmetric Time Intervals, which lead to a significant improvement in sample quality. Despite its simplicity, the proposed approach can achieve strong performance in both discrete image generation and image captioning tasks. For discrete image generation, we significantly improve previous state-of-the-art on both CIFAR-10 (which has 3K discrete 8-bit tokens) and ImageNet-64x64 (which has 12K discrete 8-bit tokens), outperforming the best autoregressive model in both sample quality (measured by FID) and efficiency. For image captioning on MS-COCO dataset, our approach achieves competitive results compared to autoregressive models.