Neural Entropy

TL;DR

This paper introduces neural entropy, a measure linking deep learning and information theory through diffusion models, revealing how neural networks efficiently compress structured data during diffusion processes.

Contribution

It defines neural entropy as a new metric to quantify information in neural networks during diffusion, connecting data distribution, diffusion process, and information compression.

Findings

Neural entropy relates to total entropy produced by diffusion.

Diffusion models efficiently compress large data ensembles.

Neural entropy depends on data distribution and diffusion process.

Abstract

We explore the connection between deep learning and information theory through the paradigm of diffusion models. A diffusion model converts noise into structured data by reinstating, imperfectly, information that is erased when data was diffused to noise. This information is stored in a neural network during training. We quantify this information by introducing a measure called neural entropy, which is related to the total entropy produced by diffusion. Neural entropy is a function of not just the data distribution, but also the diffusive process itself. Measurements of neural entropy on a few simple image diffusion models reveal that they are extremely efficient at compressing large ensembles of structured data.