Derivative-Free Guidance in Continuous and Discrete Diffusion Models with Soft Value-Based Decoding

TL;DR

This paper introduces a novel derivative-free guidance method for diffusion models that integrates soft value functions into the sampling process, enabling optimization of reward functions without fine-tuning or differentiable proxies, applicable to various domains.

Contribution

The proposed iterative sampling algorithm incorporates soft value functions into diffusion model inference, avoiding fine-tuning and enabling direct use of non-differentiable rewards across continuous and discrete models.

Findings

Effective in image, molecule, and sequence generation

Outperforms existing guidance methods

Does not require model fine-tuning or differentiable proxies

Abstract

Diffusion models excel at capturing the natural design spaces of images, molecules, DNA, RNA, and protein sequences. However, rather than merely generating designs that are natural, we often aim to optimize downstream reward functions while preserving the naturalness of these design spaces. Existing methods for achieving this goal often require ``differentiable'' proxy models (\textit{e.g.}, classifier guidance or DPS) or involve computationally expensive fine-tuning of diffusion models (\textit{e.g.}, classifier-free guidance, RL-based fine-tuning). In our work, we propose a new method to address these challenges. Our algorithm is an iterative sampling method that integrates soft value functions, which looks ahead to how intermediate noisy states lead to high rewards in the future, into the standard inference procedure of pre-trained diffusion models. Notably, our approach avoids fine-tuning generative models and eliminates the need to construct differentiable models. This enables us to (1) directly utilize non-differentiable features/reward feedback, commonly used in many scientific domains, and (2) apply our method to recent discrete diffusion models in a principled way. Finally, we demonstrate the effectiveness of our algorithm across several domains, including image generation, molecule generation, and DNA/RNA sequence generation. The code is available at \href{https://github.com/masa-ue/SVDD}{https://github.com/masa-ue/SVDD}.