Inference-Time Alignment of Diffusion Models via Evolutionary Algorithms

TL;DR

This paper presents a novel inference-time alignment method for diffusion models using evolutionary algorithms, enabling better alignment with objectives without requiring gradients or internal model access, while being more efficient.

Contribution

It introduces a black-box, gradient-free alignment framework for diffusion models that improves alignment performance and computational efficiency.

Findings

Achieves 3-35% higher ImageReward scores than existing methods.

Requires 55-76% less GPU memory and is 72-80% faster than gradient-based approaches.

Performs competitively across multiple alignment objectives on the Open Image Preferences dataset.

Abstract

Diffusion models are state-of-the-art generative models, yet their samples often fail to satisfy application objectives such as safety constraints or domain-specific validity. Existing techniques for alignment require gradients, internal model access, or large computational budgets resulting in high compute demands, or lack of support for certain objectives. In response, we introduce an inference-time alignment framework based on evolutionary algorithms. We treat diffusion models as black boxes and search their latent space to maximize alignment objectives. Given equal or less running time, our method achieves 3-35% higher ImageReward scores than gradient-free and gradient-based methods. On the Open Image Preferences dataset, our method achieves competitive results across four popular alignment objectives. In terms of computational efficiency, we require 55% to 76% less GPU memory and are 72% to 80% faster than gradient-based methods.