Half-order Fine-Tuning for Diffusion Model: A Recursive Likelihood Ratio Optimizer

TL;DR

This paper introduces the Recursive Likelihood Ratio optimizer, a novel half-order fine-tuning method for diffusion models that improves training efficiency and stability by providing unbiased, low-variance gradient estimates, validated through extensive experiments.

Contribution

The paper proposes a new half-order fine-tuning paradigm with an unbiased gradient estimator for diffusion models, addressing limitations of existing methods like RL and truncated BP.

Findings

RLR outperforms RL and truncated BP in efficiency and stability.

Theoretical analysis confirms lower variance and unbiasedness of RLR.

Experimental results show superior image and video generation quality.

Abstract

The probabilistic diffusion model (DM), generating content by inferencing through a recursive chain structure, has emerged as a powerful framework for visual generation. After pre-training on enormous data, the model needs to be properly aligned to meet requirements for downstream applications. How to efficiently align the foundation DM is a crucial task. Contemporary methods are either based on Reinforcement Learning (RL) or truncated Backpropagation (BP). However, RL and truncated BP suffer from low sample efficiency and biased gradient estimation, respectively, resulting in limited improvement or, even worse, complete training failure. To overcome the challenges, we propose the Recursive Likelihood Ratio (RLR) optimizer, a Half-Order (HO) fine-tuning paradigm for DM. The HO gradient estimator enables the computation graph rearrangement within the recursive diffusive chain, making the RLR's gradient estimator an unbiased one with lower variance than other methods. We theoretically investigate the bias, variance, and convergence of our method. Extensive experiments are conducted on image and video generation to validate the superiority of the RLR. Furthermore, we propose a novel prompt technique that is natural for the RLR to achieve a synergistic effect.