Steering Masked Discrete Diffusion Models via Discrete Denoising Posterior Prediction

TL;DR

This paper introduces Discrete Denoising Posterior Prediction (DDPP), a scalable probabilistic inference framework for steering Masked Diffusion Models (MDMs) in discrete data generation, applicable to images, text, and proteins.

Contribution

The paper proposes DDPP, a novel, simulation-free framework for steering pre-trained MDMs through probabilistic inference, enabling control over diverse non-differentiable reward functions.

Findings

Successfully steered MDMs for class-conditional image modeling

Achieved RLHF-based alignment of MDMs with text rewards

Generated protein sequences with enhanced properties validated in wet-lab experiments

Abstract

Generative modeling of discrete data underlies important applications spanning text-based agents like ChatGPT to the design of the very building blocks of life in protein sequences. However, application domains need to exert control over the generated data by steering the generative process - typically via RLHF - to satisfy a specified property, reward, or affinity metric. In this paper, we study the problem of steering Masked Diffusion Models (MDMs), a recent class of discrete diffusion models that offer a compelling alternative to traditional autoregressive models. We introduce Discrete Denoising Posterior Prediction (DDPP), a novel framework that casts the task of steering pre-trained MDMs as a problem of probabilistic inference by learning to sample from a target Bayesian posterior. Our DDPP framework leads to a family of three novel objectives that are all simulation-free, and thus scalable while applying to general non-differentiable reward functions. Empirically, we instantiate DDPP by steering MDMs to perform class-conditional pixel-level image modeling, RLHF-based alignment of MDMs using text-based rewards, and finetuning protein language models to generate more diverse secondary structures and shorter proteins. We substantiate our designs via wet-lab validation, where we observe transient expression of reward-optimized protein sequences.