R\'enyi Neural Processes

TL;DR

Rényi Neural Processes introduce a novel divergence measure to improve uncertainty modeling in neural processes, addressing prior misspecification and achieving superior performance in regression and image inpainting tasks.

Contribution

The paper proposes Rényi Neural Processes, replacing KL divergence with Rényi divergence to mitigate prior misspecification in neural processes, leading to enhanced performance.

Findings

RNPs outperform standard NPs in log-likelihoods.

RNPs show significant improvements in regression tasks.

RNPs excel in image inpainting benchmarks.

Abstract

Neural Processes (NPs) are deep probabilistic models that represent stochastic processes by conditioning their prior distributions on a set of context points. Despite their advantages in uncertainty estimation for complex distributions, NPs enforce parameterization coupling between the conditional prior model and the posterior model. We show that this coupling amounts to prior misspecification and revisit the NP objective to address this issue. More specifically, we propose R\'enyi Neural Processes (RNP), a method that replaces the standard KL divergence with the R\'enyi divergence, dampening the effects of the misspecified prior during posterior updates. We validate our approach across multiple benchmarks including regression and image inpainting tasks, and show significant performance improvements of RNPs in real-world problems. Our extensive experiments show consistently better log-likelihoods over state-of-the-art NP models.