Stochastic Control for Fine-tuning Diffusion Models: Optimality, Regularity, and Convergence

TL;DR

This paper introduces a stochastic control framework for fine-tuning diffusion models, providing theoretical guarantees and a convergent algorithm, thereby advancing the understanding and practical application of model customization.

Contribution

It develops a novel stochastic control approach with theoretical analysis and a convergent policy iteration algorithm for fine-tuning diffusion models.

Findings

PI-FT algorithm achieves global linear convergence.

The control sequences maintain regularity during training.

Framework extensions to parametric and continuous-time settings.

Abstract

Diffusion models have emerged as powerful tools for generative modeling, demonstrating exceptional capability in capturing target data distributions from large datasets. However, fine-tuning these massive models for specific downstream tasks, constraints, and human preferences remains a critical challenge. While recent advances have leveraged reinforcement learning algorithms to tackle this problem, much of the progress has been empirical, with limited theoretical understanding. To bridge this gap, we propose a stochastic control framework for fine-tuning diffusion models. Building on denoising diffusion probabilistic models as the pre-trained reference dynamics, our approach integrates linear dynamics control with Kullback-Leibler regularization. We establish the well-posedness and regularity of the stochastic control problem and develop a policy iteration algorithm (PI-FT) for numerical solution. We show that PI-FT achieves global convergence at a linear rate. Unlike existing work that assumes regularities throughout training, we prove that the control and value sequences generated by the algorithm maintain the regularity. Additionally, we explore extensions of our framework to parametric settings and continuous-time formulations, and demonstrate the practical effectiveness of the proposed PI-FT algorithm through numerical experiments. Our code is available at https://github.com/yinbinhan/fine-tuning-of-diffusion-models.