Iterative Tilting for Diffusion Fine-Tuning

TL;DR

This paper presents iterative tilting, a gradient-free approach for fine-tuning diffusion models towards reward-tilted distributions by decomposing large tilts into smaller, tractable steps validated on a Gaussian mixture example.

Contribution

The paper introduces a novel gradient-free method for diffusion model fine-tuning that avoids backpropagation through sampling chains by decomposing reward tilts into smaller steps.

Findings

Validated on a Gaussian mixture with linear reward

Achieves tractable score updates via Taylor expansion

Demonstrates effectiveness without backpropagation

Abstract

We introduce iterative tilting, a gradient-free method for fine-tuning diffusion models toward reward-tilted distributions. The method decomposes a large reward tilt $\exp(\lambda r)$ into $N$ sequential smaller tilts, each admitting a tractable score update via first-order Taylor expansion. This requires only forward evaluations of the reward function and avoids backpropagating through sampling chains. We validate on a two-dimensional Gaussian mixture with linear reward, where the exact tilted distribution is available in closed form.