Inference with Deep Generative Priors in High Dimensions

TL;DR

This paper introduces ML-VAMP, a novel algorithm for inference in deep generative models that provides exact performance predictions and can achieve Bayes-optimal estimates in high-dimensional inverse problems.

Contribution

The paper develops ML-VAMP, a new algorithm for multi-layer neural network inference with exact performance analysis and optimality guarantees in high dimensions.

Findings

ML-VAMP accurately predicts inference performance in high-dimensional limits.

ML-VAMP achieves Bayes-optimal MSE under certain conditions.

The method provides a computationally efficient solution for deep generative prior inference.

Abstract

Deep generative priors offer powerful models for complex-structured data, such as images, audio, and text. Using these priors in inverse problems typically requires estimating the input and/or hidden signals in a multi-layer deep neural network from observation of its output. While these approaches have been successful in practice, rigorous performance analysis is complicated by the non-convex nature of the underlying optimization problems. This paper presents a novel algorithm, Multi-Layer Vector Approximate Message Passing (ML-VAMP), for inference in multi-layer stochastic neural networks. ML-VAMP can be configured to compute maximum a priori (MAP) or approximate minimum mean-squared error (MMSE) estimates for these networks. We show that the performance of ML-VAMP can be exactly predicted in a certain high-dimensional random limit. Furthermore, under certain conditions, ML-VAMP yields estimates that achieve the minimum (i.e., Bayes-optimal) MSE as predicted by the replica method. In this way, ML-VAMP provides a computationally efficient method for multi-layer inference with an exact performance characterization and testable conditions for optimality in the large-system limit.