Towards a Mechanistic Explanation of Diffusion Model Generalization

TL;DR

This paper introduces a simple, training-free explanation for how diffusion models generalize, highlighting a shared local inductive bias and proposing novel denoising algorithms that mimic neural network behavior with high visual fidelity.

Contribution

It presents a new mechanistic understanding of diffusion model generalization based on local denoising operations and introduces algorithms that replicate network behavior without training.

Findings

Denoising algorithms replicate network outputs with high visual similarity.

The proposed methods outperform previous approaches in mean squared error.

Shared local inductive bias observed across various architectures.

Abstract

We propose a simple, training-free mechanism which explains the generalization behaviour of diffusion models. By comparing pre-trained diffusion models to their theoretically optimal empirical counterparts, we identify a shared local inductive bias across a variety of network architectures. From this observation, we hypothesize that network denoisers generalize through localized denoising operations, as these operations approximate the training objective well over much of the training distribution. To validate our hypothesis, we introduce novel denoising algorithms which aggregate local empirical denoisers to replicate network behaviour. Comparing these algorithms to network denoisers across forward and reverse diffusion processes, our approach exhibits consistent visual similarity to neural network outputs, with lower mean squared error than previously proposed methods.