DiffIER: Optimizing Diffusion Models with Iterative Error Reduction

TL;DR

DiffIER introduces an iterative error reduction technique to optimize diffusion model inference, addressing the training-inference gap and improving sample quality across various generative tasks.

Contribution

The paper proposes a novel plug-and-play optimization framework that reduces accumulated inference errors in diffusion models through iterative minimization, enhancing generation quality.

Findings

Outperforms baseline methods in conditional generation tasks

Effective across text-to-image, super-resolution, and speech synthesis

Reduces sensitivity to guidance weight by minimizing inference error

Abstract

Diffusion models have demonstrated remarkable capabilities in generating high-quality samples and enhancing performance across diverse domains through Classifier-Free Guidance (CFG). However, the quality of generated samples is highly sensitive to the selection of the guidance weight. In this work, we identify a critical ``training-inference gap'' and we argue that it is the presence of this gap that undermines the performance of conditional generation and renders outputs highly sensitive to the guidance weight. We quantify this gap by measuring the accumulated error during the inference stage and establish a correlation between the selection of guidance weight and minimizing this gap. Furthermore, to mitigate this gap, we propose DiffIER, an optimization-based method for high-quality generation. We demonstrate that the accumulated error can be effectively reduced by an iterative error minimization at each step during inference. By introducing this novel plug-and-play optimization framework, we enable the optimization of errors at every single inference step and enhance generation quality. Empirical results demonstrate that our proposed method outperforms baseline approaches in conditional generation tasks. Furthermore, the method achieves consistent success in text-to-image generation, image super-resolution, and text-to-speech generation, underscoring its versatility and potential for broad applications in future research.