Adjoint Matching: Fine-tuning Flow and Diffusion Generative Models with Memoryless Stochastic Optimal Control

TL;DR

This paper introduces Adjoint Matching, a novel method for reward fine-tuning of flow and diffusion generative models using stochastic optimal control, ensuring better sample quality and generalization.

Contribution

It presents a theoretically grounded approach with a specific noise schedule and a new algorithm that outperforms existing SOC methods in fine-tuning generative models.

Findings

Improved sample realism and consistency.

Enhanced generalization to unseen rewards.

Retained diversity in generated samples.

Abstract

Dynamical generative models that produce samples through an iterative process, such as Flow Matching and denoising diffusion models, have seen widespread use, but there have not been many theoretically-sound methods for improving these models with reward fine-tuning. In this work, we cast reward fine-tuning as stochastic optimal control (SOC). Critically, we prove that a very specific memoryless noise schedule must be enforced during fine-tuning, in order to account for the dependency between the noise variable and the generated samples. We also propose a new algorithm named Adjoint Matching which outperforms existing SOC algorithms, by casting SOC problems as a regression problem. We find that our approach significantly improves over existing methods for reward fine-tuning, achieving better consistency, realism, and generalization to unseen human preference reward models, while retaining sample diversity.