Alleviating Exposure Bias in Diffusion Models through Sampling with Shifted Time Steps

TL;DR

This paper introduces a novel sampling method called Time-Shift Sampler that alleviates exposure bias in diffusion models without retraining, leading to improved image synthesis quality with fewer sampling steps.

Contribution

The paper proposes a new sampling technique, Time-Shift Sampler, that reduces exposure bias in diffusion probabilistic models without retraining, enhancing efficiency and quality.

Findings

Significant improvements in FID scores across datasets and sampling methods.

The method outperforms vanilla DDIM with fewer sampling steps.

The approach integrates seamlessly with existing algorithms.

Abstract

Diffusion Probabilistic Models (DPM) have shown remarkable efficacy in the synthesis of high-quality images. However, their inference process characteristically requires numerous, potentially hundreds, of iterative steps, which could exaggerate the problem of exposure bias due to the training and inference discrepancy. Previous work has attempted to mitigate this issue by perturbing inputs during training, which consequently mandates the retraining of the DPM. In this work, we conduct a systematic study of exposure bias in DPM and, intriguingly, we find that the exposure bias could be alleviated with a novel sampling method that we propose, without retraining the model. We empirically and theoretically show that, during inference, for each backward time step $t$ and corresponding state $\hat{x}_t$, there might exist another time step $t_s$ which exhibits superior coupling with $\hat{x}_t$. Based on this finding, we introduce a sampling method named Time-Shift Sampler. Our framework can be seamlessly integrated to existing sampling algorithms, such as DDPM, DDIM and other high-order solvers, inducing merely minimal additional computations. Experimental results show our method brings significant and consistent improvements in FID scores on different datasets and sampling methods. For example, integrating Time-Shift Sampler to F-PNDM yields a FID=3.88, achieving 44.49\% improvements as compared to F-PNDM, on CIFAR-10 with 10 sampling steps, which is more performant than the vanilla DDIM with 100 sampling steps. Our code is available at https://github.com/Mingxiao-Li/TS-DPM.