Large Deviation Methods for Approximate Probabilistic Inference

TL;DR

This paper introduces large deviation techniques to compute bounds on marginal probabilities in large belief networks with binary variables, enabling approximate inference where exact methods are infeasible.

Contribution

It develops a novel approach using large deviation theory to derive rigorous bounds and convergence rates for probabilistic inference in large, complex networks.

Findings

Provides bounds on marginal probabilities in large networks

Establishes convergence rates for bounds as network size increases

Applicable to networks with sigmoid and noisy-OR transfer functions

Abstract

We study two-layer belief networks of binary random variables in which the conditional probabilities Pr[childlparents] depend monotonically on weighted sums of the parents. In large networks where exact probabilistic inference is intractable, we show how to compute upper and lower bounds on many probabilities of interest. In particular, using methods from large deviation theory, we derive rigorous bounds on marginal probabilities such as Pr[children] and prove rates of convergence for the accuracy of our bounds as a function of network size. Our results apply to networks with generic transfer function parameterizations of the conditional probability tables, such as sigmoid and noisy-OR. They also explicitly illustrate the types of averaging behavior that can simplify the problem of inference in large networks.