Maximal Information Divergence from Statistical Models defined by Neural Networks

TL;DR

This paper reviews recent findings on the maximum Kullback-Leibler divergence from neural network-based statistical models, providing methods to compute these divergences and presenting new results for specific deep belief networks.

Contribution

It offers a comprehensive review of maximal divergence values for various neural network models and introduces new results for deep and narrow belief networks.

Findings

Maximal divergence values are characterized for several neural network models.

Methods to compute divergence from simple sub- or super-models are illustrated.

New results are provided for deep and narrow belief networks with finite units.

Abstract

We review recent results about the maximal values of the Kullback-Leibler information divergence from statistical models defined by neural networks, including naive Bayes models, restricted Boltzmann machines, deep belief networks, and various classes of exponential families. We illustrate approaches to compute the maximal divergence from a given model starting from simple sub- or super-models. We give a new result for deep and narrow belief networks with finite-valued units.