Bayesian Inference with Nonlinear Generative Models: Comments on Secure Learning

TL;DR

This paper investigates the secrecy potential of nonlinear generative models using the replica method, revealing a phase transition in Bayesian inference and proposing a secure coding scheme that achieves the secrecy capacity of the wiretap channel.

Contribution

It introduces a novel analysis of nonlinear generative models' secrecy properties and proposes a secure coding scheme based on phase transition insights.

Findings

Nonlinear models exhibit an all-or-nothing phase transition in inference accuracy.

A new secure coding scheme achieves the wiretap channel's secrecy capacity.

Strictly nonlinear models can be perfectly secured without additional secure coding.

Abstract

Unlike the classical linear model, nonlinear generative models have been addressed sparsely in the literature of statistical learning. This work aims to bringing attention to these models and their secrecy potential. To this end, we invoke the replica method to derive the asymptotic normalized cross entropy in an inverse probability problem whose generative model is described by a Gaussian random field with a generic covariance function. Our derivations further demonstrate the asymptotic statistical decoupling of the Bayesian estimator and specify the decoupled setting for a given nonlinear model. The replica solution depicts that strictly nonlinear models establish an all-or-nothing phase transition: There exists a critical load at which the optimal Bayesian inference changes from perfect to an uncorrelated learning. Based on this finding, we design a new secure coding scheme which achieves the secrecy capacity of the wiretap channel. This interesting result implies that strictly nonlinear generative models are perfectly secured without any secure coding. We justify this latter statement through the analysis of an illustrative model for perfectly secure and reliable inference.