Understanding Generalization in Diffusion Distillation via Probability Flow Distance

TL;DR

This paper introduces probability flow distance (PFD), a practical and theoretically grounded metric for measuring generalization in diffusion distillation models, revealing key behaviors like scaling, double descent, and bias-variance dynamics.

Contribution

The work proposes PFD as a new metric for evaluating generalization in diffusion distillation, connecting theoretical insights with empirical observations.

Findings

Quantitative scaling from memorization to generalization

Epoch-wise double descent training dynamics

Bias-variance decomposition in diffusion models

Abstract

Diffusion distillation provides an effective approach for learning lightweight and few-steps diffusion models with efficient generation. However, evaluating their generalization remains challenging: theoretical metrics are often impractical for high-dimensional data, while no practical metrics rigorously measure generalization. In this work, we bridge this gap by introducing probability flow distance (\texttt{PFD}), a theoretically grounded and computationally efficient metric to measure generalization. Specifically, \texttt{PFD} quantifies the distance between distributions by comparing their noise-to-data mappings induced by the probability flow ODE. Using \texttt{PFD} under the diffusion distillation setting, we empirically uncover several key generalization behaviors, including: (1) quantitative scaling behavior from memorization to generalization, (2) epoch-wise double descent training dynamics, and (3) bias-variance decomposition. Beyond these insights, our work lays a foundation for generalization studies in diffusion distillation and bridges them with diffusion training.