Alignment of Diffusion Model and Flow Matching for Text-to-Image Generation

TL;DR

This paper introduces a novel, computationally efficient framework for aligning diffusion and flow matching models in text-to-image generation by leveraging reward-weighted distributions, avoiding extensive fine-tuning.

Contribution

It proposes a finetuning-free guidance network for diffusion models and a training-free approach for flow matching, enhancing generation quality with reduced computational costs.

Findings

Achieves comparable performance to finetuning methods with 60% less computation.

Introduces a guidance network for diffusion models that reduces artifacts.

Provides a training-free method for flow matching that improves image quality.

Abstract

Diffusion models and flow matching have demonstrated remarkable success in text-to-image generation. While many existing alignment methods primarily focus on fine-tuning pre-trained generative models to maximize a given reward function, these approaches require extensive computational resources and may not generalize well across different objectives. In this work, we propose a novel alignment framework by leveraging the underlying nature of the alignment problem -- sampling from reward-weighted distributions -- and show that it applies to both diffusion models (via score guidance) and flow matching models (via velocity guidance). The score function (velocity field) required for the reward-weighted distribution can be decomposed into the pre-trained score (velocity field) plus a conditional expectation of the reward. For the alignment on the diffusion model, we identify a fundamental challenge: the adversarial nature of the guidance term can introduce undesirable artifacts in the generated images. Therefore, we propose a finetuning-free framework that trains a guidance network to estimate the conditional expectation of the reward. We achieve comparable performance to finetuning-based models with one-step generation with at least a 60% reduction in computational cost. For the alignment on flow matching, we propose a training-free framework that improves the generation quality without additional computational cost.