Improving the Training of Rectified Flows

TL;DR

This paper introduces techniques to enhance rectified flows for generative modeling, enabling high-quality image synthesis with fewer function evaluations by training in a single iteration, thus reducing computational costs.

Contribution

It demonstrates that a single iteration of the Reflow algorithm suffices for learning nearly straight trajectories, and proposes methods like U-shaped timestep distribution and LPIPS-Huber premetric to improve one-round training.

Findings

Up to 75% FID improvement on CIFAR-10 with 1 NFE

Outperforms state-of-the-art distillation methods on ImageNet 64x64

Rivals the performance of improved consistency training (iCT)

Abstract

Diffusion models have shown great promise for image and video generation, but sampling from state-of-the-art models requires expensive numerical integration of a generative ODE. One approach for tackling this problem is rectified flows, which iteratively learn smooth ODE paths that are less susceptible to truncation error. However, rectified flows still require a relatively large number of function evaluations (NFEs). In this work, we propose improved techniques for training rectified flows, allowing them to compete with \emph{knowledge distillation} methods even in the low NFE setting. Our main insight is that under realistic settings, a single iteration of the Reflow algorithm for training rectified flows is sufficient to learn nearly straight trajectories; hence, the current practice of using multiple Reflow iterations is unnecessary. We thus propose techniques to improve one-round training of rectified flows, including a U-shaped timestep distribution and LPIPS-Huber premetric. With these techniques, we improve the FID of the previous 2-rectified flow by up to 75\% in the 1 NFE setting on CIFAR-10. On ImageNet 64$\times$64, our improved rectified flow outperforms the state-of-the-art distillation methods such as consistency distillation and progressive distillation in both one-step and two-step settings and rivals the performance of improved consistency training (iCT) in FID. Code is available at https://github.com/sangyun884/rfpp.