Structured Uncertainty Prediction Networks

TL;DR

This paper introduces a novel neural network that predicts full Gaussian covariance matrices for image reconstructions, enabling better modeling of structured uncertainties and applications like denoising.

Contribution

It is the first to predict full covariance matrices in image synthesis, improving uncertainty modeling over previous diagonal-only approaches.

Findings

Accurately reconstructs correlated residual distributions in synthetic data.

Generates plausible high-frequency samples for face images.

Uses predicted covariances for structure-preserving denoising.

Abstract

This paper is the first work to propose a network to predict a structured uncertainty distribution for a synthesized image. Previous approaches have been mostly limited to predicting diagonal covariance matrices. Our novel model learns to predict a full Gaussian covariance matrix for each reconstruction, which permits efficient sampling and likelihood evaluation. We demonstrate that our model can accurately reconstruct ground truth correlated residual distributions for synthetic datasets and generate plausible high frequency samples for real face images. We also illustrate the use of these predicted covariances for structure preserving image denoising.