CARD: Classification and Regression Diffusion Models

TL;DR

CARD models leverage diffusion-based generative techniques combined with mean estimators to accurately predict complex, multi-modal conditional distributions and improve uncertainty quantification in supervised learning tasks.

Contribution

Introduction of CARD models that integrate diffusion generative models with pre-trained mean estimators for enhanced distribution prediction and uncertainty estimation.

Findings

CARD outperforms state-of-the-art methods in distribution prediction.

CARD effectively captures multi-modal conditional distributions.

The stochastic outputs enable finer confidence assessment.

Abstract

Learning the distribution of a continuous or categorical response variable $\boldsymbol y$ given its covariates $\boldsymbol x$ is a fundamental problem in statistics and machine learning. Deep neural network-based supervised learning algorithms have made great progress in predicting the mean of $\boldsymbol y$ given $\boldsymbol x$, but they are often criticized for their ability to accurately capture the uncertainty of their predictions. In this paper, we introduce classification and regression diffusion (CARD) models, which combine a denoising diffusion-based conditional generative model and a pre-trained conditional mean estimator, to accurately predict the distribution of $\boldsymbol y$ given $\boldsymbol x$. We demonstrate the outstanding ability of CARD in conditional distribution prediction with both toy examples and real-world datasets, the experimental results on which show that CARD in general outperforms state-of-the-art methods, including Bayesian neural network-based ones that are designed for uncertainty estimation, especially when the conditional distribution of $\boldsymbol y$ given $\boldsymbol x$ is multi-modal. In addition, we utilize the stochastic nature of the generative model outputs to obtain a finer granularity in model confidence assessment at the instance level for classification tasks.