Towards understanding Diffusion Models (on Graphs)

TL;DR

This paper provides an overview and empirical analysis of diffusion models, focusing on their mathematical foundations, role of noise, sampling methods, and neural network functions, with the goal of advancing their application in graph machine learning.

Contribution

It offers a comprehensive overview of diffusion models' theoretical perspectives and conducts experiments to understand their core mechanisms in simplified settings.

Findings

Noise significantly influences model behavior

Sampling method choice impacts outcomes

Neural network complexity may not be essential for performance

Abstract

Diffusion models have emerged from various theoretical and methodological perspectives, each offering unique insights into their underlying principles. In this work, we provide an overview of the most prominent approaches, drawing attention to their striking analogies -- namely, how seemingly diverse methodologies converge to a similar mathematical formulation of the core problem. While our ultimate goal is to understand these models in the context of graphs, we begin by conducting experiments in a simpler setting to build foundational insights. Through an empirical investigation of different diffusion and sampling techniques, we explore three critical questions: (1) What role does noise play in these models? (2) How significantly does the choice of the sampling method affect outcomes? (3) What function is the neural network approximating, and is high complexity necessary for optimal performance? Our findings aim to enhance the understanding of diffusion models and in the long run their application in graph machine learning.