An analytic theory of creativity in convolutional diffusion models

TL;DR

This paper develops an analytic, interpretable theory explaining how convolutional diffusion models generate highly original images through local and equivariant biases, revealing a patch mosaic mechanism of creativity.

Contribution

It introduces a mechanistic local score model that predicts diffusion model outputs and uncovers a patch mosaic process underlying creativity in image generation.

Findings

High predictive accuracy of the model on various datasets (median r^2 > 0.94)

Reveals a local patch mosaic mechanism for creativity in diffusion models

Partially predicts outputs of attention-enabled UNets, suggesting attention's role in semantic coherence.

Abstract

We obtain an analytic, interpretable and predictive theory of creativity in convolutional diffusion models. Indeed, score-matching diffusion models can generate highly original images that lie far from their training data. However, optimal score-matching theory suggests that these models should only be able to produce memorized training examples. To reconcile this theory-experiment gap, we identify two simple inductive biases, locality and equivariance, that: (1) induce a form of combinatorial creativity by preventing optimal score-matching; (2) result in fully analytic, completely mechanistically interpretable, local score (LS) and equivariant local score (ELS) machines that, (3) after calibrating a single time-dependent hyperparameter can quantitatively predict the outputs of trained convolution only diffusion models (like ResNets and UNets) with high accuracy (median $r^2$ of $0.95, 0.94, 0.94, 0.96$ for our top model on CIFAR10, FashionMNIST, MNIST, and CelebA). Our model reveals a locally consistent patch mosaic mechanism of creativity, in which diffusion models create exponentially many novel images by mixing and matching different local training set patches at different scales and image locations. Our theory also partially predicts the outputs of pre-trained self-attention enabled UNets (median $r^2 \sim 0.77$ on CIFAR10), revealing an intriguing role for attention in carving out semantic coherence from local patch mosaics.