Unmasking Bias in Diffusion Model Training

TL;DR

This paper identifies bias in the training of diffusion models as a key factor in their slow convergence and color shift issues, and proposes a new loss weighting strategy to improve sample quality and training efficiency.

Contribution

It provides theoretical analysis of bias in diffusion model training and introduces a simple, effective weighting strategy to mitigate this bias.

Findings

Enhanced sample quality with the new weighting strategy

Improved training and sampling efficiency

Systematic analysis of bias impact in diffusion models

Abstract

Denoising diffusion models have emerged as a dominant approach for image generation, however they still suffer from slow convergence in training and color shift issues in sampling. In this paper, we identify that these obstacles can be largely attributed to bias and suboptimality inherent in the default training paradigm of diffusion models. Specifically, we offer theoretical insights that the prevailing constant loss weight strategy in $\epsilon$-prediction of diffusion models leads to biased estimation during the training phase, hindering accurate estimations of original images. To address the issue, we propose a simple but effective weighting strategy derived from the unlocked biased part. Furthermore, we conduct a comprehensive and systematic exploration, unraveling the inherent bias problem in terms of its existence, impact and underlying reasons. These analyses contribute to advancing the understanding of diffusion models. Empirical results demonstrate that our method remarkably elevates sample quality and displays improved efficiency in both training and sampling processes, by only adjusting loss weighting strategy. The code is released publicly at \url{https://github.com/yuhuUSTC/Debias}