Finite Difference Flow Optimization for RL Post-Training of Text-to-Image Models

TL;DR

This paper introduces an online reinforcement learning method for post-training optimization of text-to-image diffusion models, reducing update variance and improving image quality and prompt alignment.

Contribution

It proposes a novel RL approach that considers the entire sampling process as a single action, leading to faster convergence and better results.

Findings

Faster convergence compared to previous methods

Higher image quality and prompt alignment

Effective with various reward metrics

Abstract

Reinforcement learning (RL) has become a standard technique for post-training diffusion-based image synthesis models, as it enables learning from reward signals to explicitly improve desirable aspects such as image quality and prompt alignment. In this paper, we propose an online RL variant that reduces the variance in the model updates by sampling paired trajectories and pulling the flow velocity in the direction of the more favorable image. Unlike existing methods that treat each sampling step as a separate policy action, we consider the entire sampling process as a single action. We experiment with both high-quality vision language models and off-the-shelf quality metrics for rewards, and evaluate the outputs using a broad set of metrics. Our method converges faster and yields higher output quality and prompt alignment than previous approaches.