Bridging SFT and DPO for Diffusion Model Alignment with Self-Sampling Preference Optimization

TL;DR

This paper introduces SSPO, a novel alignment method combining the stability of supervised fine-tuning with the generalization of reinforcement learning, using self-sampling and checkpoint replay to improve diffusion model training.

Contribution

The paper proposes SSPO, a new alignment technique that integrates checkpoint replay and self-sampling regularization to enhance diffusion model training without paired data or reward models.

Findings

SSPO outperforms existing methods on text-to-image benchmarks.

SSPO demonstrates strong performance on text-to-video tasks.

The approach effectively balances training stability and generalization.

Abstract

Existing post-training techniques are broadly categorized into supervised fine-tuning (SFT) and reinforcement learning (RL) methods; the former is stable during training but suffers from limited generalization, while the latter, despite its stronger generalization capability, relies on additional preference data or reward models and carries the risk of reward exploitation. In order to preserve the advantages of both SFT and RL -- namely, eliminating the need for paired data and reward models while retaining the training stability of SFT and the generalization ability of RL -- a new alignment method, Self-Sampling Preference Optimization (SSPO), is proposed in this paper. SSPO introduces a Random Checkpoint Replay (RCR) strategy that utilizes historical checkpoints to construct paired data, thereby effectively mitigating overfitting. Simultaneously, a Self-Sampling Regularization (SSR) strategy is employed to dynamically evaluate the quality of generated samples; when the generated samples are more likely to be winning samples, the approach automatically switches from DPO (Direct Preference Optimization) to SFT, ensuring that the training process accurately reflects the quality of the samples. Experimental results demonstrate that SSPO not only outperforms existing methods on text-to-image benchmarks, but its effectiveness has also been validated in text-to-video tasks. We validate SSPO across both text-to-image and text-to-video benchmarks. SSPO surpasses all previous approaches on the text-to-image benchmarks and demonstrates outstanding performance on the text-to-video benchmarks.