Rethinking Timesteps Samplers and Prediction Types

TL;DR

This paper investigates the training challenges of diffusion models under limited resources, identifying key issues related to loss variation across timesteps and prediction types, and proposes mixed-prediction strategies to improve training efficiency.

Contribution

It introduces an analysis of training difficulties with limited resources and proposes mixed-prediction approaches to enhance diffusion model training.

Findings

Loss variation across timesteps disrupts training progress.

Different prediction types have varying effectiveness depending on the task.

Mixed-prediction strategies could improve training stability and efficiency.

Abstract

Diffusion models suffer from the huge consumption of time and resources to train. For example, diffusion models need hundreds of GPUs to train for several weeks for a high-resolution generative task to meet the requirements of an extremely large number of iterations and a large batch size. Training diffusion models become a millionaire's game. With limited resources that only fit a small batch size, training a diffusion model always fails. In this paper, we investigate the key reasons behind the difficulties of training diffusion models with limited resources. Through numerous experiments and demonstrations, we identified a major factor: the significant variation in the training losses across different timesteps, which can easily disrupt the progress made in previous iterations. Moreover, different prediction types of $x_0$ exhibit varying effectiveness depending on the task and timestep. We hypothesize that using a mixed-prediction approach to identify the most accurate $x_0$ prediction type could potentially serve as a breakthrough in addressing this issue. In this paper, we outline several challenges and insights, with the hope of inspiring further research aimed at tackling the limitations of training diffusion models with constrained resources, particularly for high-resolution tasks.