Loss Functions in Diffusion Models: A Comparative Study

TL;DR

This paper provides a comprehensive comparison of loss functions used in diffusion models, unifying various objectives under a common framework and empirically analyzing their performance impacts.

Contribution

It offers a systematic theoretical overview and empirical evaluation of different loss functions in diffusion models, clarifying their relationships and effects on model performance.

Findings

Loss functions can be unified under the variational lower bound framework.

Different objectives influence the quality and likelihood estimation capabilities.

Empirical results highlight conditions where objectives diverge in performance.

Abstract

Diffusion models have emerged as powerful generative models, inspiring extensive research into their underlying mechanisms. One of the key questions in this area is the loss functions these models shall train with. Multiple formulations have been introduced in the literature over the past several years with some links and some critical differences stemming from various initial considerations. In this paper, we explore the different target objectives and corresponding loss functions in detail. We present a systematic overview of their relationships, unifying them under the framework of the variational lower bound objective. We complement this theoretical analysis with an empirical study providing insights into the conditions under which these objectives diverge in performance and the underlying factors contributing to such deviations. Additionally, we evaluate how the choice of objective impacts the model ability to achieve specific goals, such as generating high-quality samples or accurately estimating likelihoods. This study offers a unified understanding of loss functions in diffusion models, contributing to more efficient and goal-oriented model designs in future research.