Sampling Binary Data by Denoising through Score Functions

TL;DR

This paper extends score-based generative modeling to binary data by using Bernoulli noise, deriving a denoising formula, and proposing a Langevin-like sampler, enabling effective sampling and denoising of binary distributions.

Contribution

It introduces a TMF-like denoising framework for binary data with Bernoulli noise and develops a practical sampling method without continuous stochastic processes.

Findings

Sampling becomes easier at high Bernoulli noise levels.

The proposed method effectively denoises and samples binary data.

Theoretical analysis confirms the sampler's efficiency across noise levels.

Abstract

Gaussian smoothing combined with a probabilistic framework for denoising via the empirical Bayes formalism, i.e., the Tweedie-Miyasawa formula (TMF), are the two key ingredients in the success of score-based generative models in Euclidean spaces. Smoothing holds the key for easing the problem of learning and sampling in high dimensions, denoising is needed for recovering the original signal, and TMF ties these together via the score function of noisy data. In this work, we extend this paradigm to the problem of learning and sampling the distribution of binary data on the Boolean hypercube by adopting Bernoulli noise, instead of Gaussian noise, as a smoothing device. We first derive a TMF-like expression for the optimal denoiser for the Hamming loss, where a score function naturally appears. Sampling noisy binary data is then achieved using a Langevin-like sampler which we theoretically analyze for different noise levels. At high Bernoulli noise levels sampling becomes easy, akin to log-concave sampling in Euclidean spaces. In addition, we extend the sequential multi-measurement sampling of Saremi et al. (2024) to the binary setting where we can bring the "effective noise" down by sampling multiple noisy measurements at a fixed noise level, without the need for continuous-time stochastic processes. We validate our formalism and theoretical findings by experiments on synthetic data and binarized images.