Constrained Diffusion with Trust Sampling

TL;DR

This paper introduces Trust Sampling, a novel method for constrained diffusion that balances model fidelity and constraint adherence through optimization and trust estimation, improving generation quality across domains.

Contribution

It proposes a training-free, optimization-based constrained diffusion approach that uses trust estimation to enhance generation accuracy and flexibility.

Findings

Significant improvements in image and 3D motion generation quality.

Effective balancing between diffusion model fidelity and constraint satisfaction.

Versatile application across diverse generative tasks.

Abstract

Diffusion models have demonstrated significant promise in various generative tasks; however, they often struggle to satisfy challenging constraints. Our approach addresses this limitation by rethinking training-free loss-guided diffusion from an optimization perspective. We formulate a series of constrained optimizations throughout the inference process of a diffusion model. In each optimization, we allow the sample to take multiple steps along the gradient of the proxy constraint function until we can no longer trust the proxy, according to the variance at each diffusion level. Additionally, we estimate the state manifold of diffusion model to allow for early termination when the sample starts to wander away from the state manifold at each diffusion step. Trust sampling effectively balances between following the unconditional diffusion model and adhering to the loss guidance, enabling more flexible and accurate constrained generation. We demonstrate the efficacy of our method through extensive experiments on complex tasks, and in drastically different domains of images and 3D motion generation, showing significant improvements over existing methods in terms of generation quality. Our implementation is available at https://github.com/will-s-h/trust-sampling.