CountsDiff: A Diffusion Model on the Natural Numbers for Generation and Imputation of Count-Based Data

TL;DR

CountsDiff is a novel diffusion model tailored for count-based data, extending existing frameworks with new features and demonstrating strong performance on image and biological datasets.

Contribution

It introduces a direct parameterization and new features like continuous-time training and non-monotone reverse dynamics for count data modeling.

Findings

CountsDiff matches or surpasses state-of-the-art models in RNA-seq imputation.

The model performs well on natural image datasets like CIFAR-10 and CelebA.

Design parameters significantly influence model performance and flexibility.

Abstract

Diffusion models have excelled at generative tasks for both continuous and token-based domains, but their application to discrete ordinal data remains underdeveloped. We present CountsDiff, a diffusion framework designed to natively model distributions on the natural numbers. CountsDiff extends the Blackout diffusion framework by simplifying its formulation through a direct parameterization in terms of a survival probability schedule and an explicit loss weighting. This introduces flexibility through design parameters with direct analogues in existing diffusion modeling frameworks. Beyond this reparameterization, CountsDiff introduces features from modern diffusion models, previously absent in counts-based domains, including continuous-time training, classifier-free guidance, and churn/remasking reverse dynamics that allow non-monotone reverse trajectories. We propose an initial instantiation of CountsDiff and validate it on natural image datasets (CIFAR-10, CelebA), exploring the effects of varying the introduced design parameters in a complex, well-studied, and interpretable data domain. We then highlight biological count assays as a natural use case, evaluating CountsDiff on single-cell RNA-seq imputation in a fetal cell and heart cell atlas. Remarkably, we find that even this simple instantiation matches or surpasses the performance of a state-of-the-art discrete generative model and leading RNA-seq imputation methods, while leaving substantial headroom for further gains through optimized design choices in future work.