Psi-Sampler: Initial Particle Sampling for SMC-Based Inference-Time Reward Alignment in Score Models

TL;DR

Psi-Sampler introduces a novel SMC-based framework with pCNL initialization for more effective inference-time reward alignment in score-based generative models, improving sampling efficiency and performance across various tasks.

Contribution

It proposes the pCNL algorithm for high-dimensional posterior sampling and demonstrates its effectiveness in reward alignment tasks, advancing score-based generative modeling.

Findings

Improved reward alignment performance in experiments

Efficient sampling in high-dimensional latent spaces

Consistent performance gains across multiple tasks

Abstract

We introduce $\Psi$-Sampler, an SMC-based framework incorporating pCNL-based initial particle sampling for effective inference-time reward alignment with a score-based generative model. Inference-time reward alignment with score-based generative models has recently gained significant traction, following a broader paradigm shift from pre-training to post-training optimization. At the core of this trend is the application of Sequential Monte Carlo (SMC) to the denoising process. However, existing methods typically initialize particles from the Gaussian prior, which inadequately captures reward-relevant regions and results in reduced sampling efficiency. We demonstrate that initializing from the reward-aware posterior significantly improves alignment performance. To enable posterior sampling in high-dimensional latent spaces, we introduce the preconditioned Crank-Nicolson Langevin (pCNL) algorithm, which combines dimension-robust proposals with gradient-informed dynamics. This approach enables efficient and scalable posterior sampling and consistently improves performance across various reward alignment tasks, including layout-to-image generation, quantity-aware generation, and aesthetic-preference generation, as demonstrated in our experiments. Project Webpage: https://psi-sampler.github.io/