Generalization Bound for Diffusion Models using Random Features

TL;DR

This paper introduces an interpretable diffusion-inspired deep random feature model that achieves comparable performance to neural networks, providing new theoretical generalization bounds and validating results on image and audio data.

Contribution

It develops a diffusion model-inspired deep random feature approach with theoretical generalization bounds, bridging interpretability and performance in complex tasks.

Findings

Comparable results to neural networks with same parameters

Derived new generalization bounds for random feature models

Validated on fashion MNIST and audio datasets

Abstract

Diffusion probabilistic models have been successfully used to generate data from noise. However, most diffusion models are computationally expensive and difficult to interpret with a lack of theoretical justification. Random feature models on the other hand have gained popularity due to their interpretability but their application to complex machine learning tasks remains limited. In this work, we present a diffusion model-inspired deep random feature model that is interpretable and gives comparable numerical results to a fully connected neural network having the same number of trainable parameters. Specifically, we extend existing results for random features and derive generalization bounds between the distribution of sampled data and the true distribution using properties of score matching. We validate our findings by generating samples on the fashion MNIST dataset and instrumental audio data.