Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks

TL;DR

This paper introduces a theoretical framework for approximating ratio-based functionals in diffusion models using SignReLU neural networks, providing convergence guarantees and bounds on generative quality.

Contribution

It develops a SignReLU neural network approach for ratio functional approximation in diffusion models, with proven convergence rates and generalization bounds.

Findings

Approximation bounds for ratio functionals using SignReLU networks

Construction of a neural estimator for the reverse diffusion process

Bounds on excess KL divergence between generated and true data distributions

Abstract

Motivated by challenges in conditional generative modeling, where the target conditional density takes the form of a ratio f1 over f2, this paper develops a theoretical framework for approximating such ratio-type functionals. Here, f1 and f2 are kernel-based marginal densities that capture structured interactions, a setting central to diffusion-based generative models. We provide a concise proof for approximating these ratio-type functionals using deep neural networks with the SignReLU activation function, leveraging the activation's piecewise structure. Under standard regularity assumptions, we establish L^p(Omega) approximation bounds and convergence rates. Specializing to Denoising Diffusion Probabilistic Models (DDPMs), we construct a SignReLU-based neural estimator for the reverse process and derive bounds on the excess Kullback-Leibler (KL) risk between the generated and true data distributions. Our analysis decomposes this excess risk into approximation and estimation error components. These results provide generalization guarantees for finite-sample training of diffusion-based generative models.