Improved Techniques for Maximum Likelihood Estimation for Diffusion ODEs

TL;DR

This paper introduces advanced techniques for maximum likelihood estimation in diffusion ODEs, significantly improving likelihood scores on image datasets by optimizing training and evaluation methods.

Contribution

It proposes velocity parameterization, variance reduction, high-order flow matching, and a novel dequantization method to enhance diffusion ODE likelihood estimation.

Findings

Achieved state-of-the-art likelihood scores on CIFAR-10 and ImageNet-32.

Improved training convergence and likelihood accuracy without data augmentation.

Introduced a training-free dequantization method for diffusion models.

Abstract

Diffusion models have exhibited excellent performance in various domains. The probability flow ordinary differential equation (ODE) of diffusion models (i.e., diffusion ODEs) is a particular case of continuous normalizing flows (CNFs), which enables deterministic inference and exact likelihood evaluation. However, the likelihood estimation results by diffusion ODEs are still far from those of the state-of-the-art likelihood-based generative models. In this work, we propose several improved techniques for maximum likelihood estimation for diffusion ODEs, including both training and evaluation perspectives. For training, we propose velocity parameterization and explore variance reduction techniques for faster convergence. We also derive an error-bounded high-order flow matching objective for finetuning, which improves the ODE likelihood and smooths its trajectory. For evaluation, we propose a novel training-free truncated-normal dequantization to fill the training-evaluation gap commonly existing in diffusion ODEs. Building upon these techniques, we achieve state-of-the-art likelihood estimation results on image datasets (2.56 on CIFAR-10, 3.43/3.69 on ImageNet-32) without variational dequantization or data augmentation, and 2.42 on CIFAR-10 with data augmentation. Code is available at \url{https://github.com/thu-ml/i-DODE}.