On the Collapse Errors Induced by the Deterministic Sampler for Diffusion Models

TL;DR

This paper uncovers collapse errors in deterministic diffusion model samplers, showing they cause data concentration issues, and analyzes their causes and potential mitigation strategies through extensive empirical experiments.

Contribution

It introduces the concept of collapse errors in ODE-based diffusion sampling and provides empirical evidence and analysis of their causes and effects.

Findings

Collapse errors cause data over-concentration in local space.

Score learning in low noise regimes affects high noise regimes adversely.

Existing techniques can mitigate some collapse errors.

Abstract

Despite the widespread adoption of deterministic samplers in diffusion models (DMs), their potential limitations remain largely unexplored. In this paper, we identify collapse errors, a previously unrecognized phenomenon in ODE-based diffusion sampling, where the sampled data is overly concentrated in local data space. To quantify this effect, we introduce a novel metric and demonstrate that collapse errors occur across a variety of settings. When investigating its underlying causes, we observe a see-saw effect, where score learning in low noise regimes adversely impacts the one in high noise regimes. This misfitting in high noise regimes, coupled with the dynamics of deterministic samplers, ultimately causes collapse errors. Guided by these insights, we apply existing techniques from sampling, training, and architecture to empirically support our explanation of collapse errors. This work provides intensive empirical evidence of collapse errors in ODE-based diffusion sampling, emphasizing the need for further research into the interplay between score learning and deterministic sampling, an overlooked yet fundamental aspect of diffusion models.