Measuring Semantic Information Production in Generative Diffusion Models

TL;DR

This paper introduces an information-theoretic method to identify when semantic features are formed during the diffusion process in generative models, revealing class-specific timing differences.

Contribution

It proposes a novel Bayesian classifier-based approach to measure semantic decision times in diffusion models, validated on Gaussian mixtures and CIFAR10.

Findings

Semantic information peaks at intermediate diffusion stages.

Different classes show distinct timing in semantic decision-making.

Semantic features emerge and vanish during the diffusion process.

Abstract

It is well known that semantic and structural features of the generated images emerge at different times during the reverse dynamics of diffusion, a phenomenon that has been connected to physical phase transitions in magnets and other materials. In this paper, we introduce a general information-theoretic approach to measure when these class-semantic "decisions" are made during the generative process. By using an online formula for the optimal Bayesian classifier, we estimate the conditional entropy of the class label given the noisy state. We then determine the time intervals corresponding to the highest information transfer between noisy states and class labels using the time derivative of the conditional entropy. We demonstrate our method on one-dimensional Gaussian mixture models and on DDPM models trained on the CIFAR10 dataset. As expected, we find that the semantic information transfer is highest in the intermediate stages of diffusion while vanishing during the final stages. However, we found sizable differences between the entropy rate profiles of different classes, suggesting that different "semantic decisions" are located at different intermediate times.