Correcting beliefs in the mean-field and Bethe approximations using linear response

TL;DR

This paper introduces an extended method for correcting beliefs in mean-field and Bethe approximations by incorporating response matrix information, significantly improving marginal estimates especially in weakly coupled and frustrated models.

Contribution

It extends a recent variational free energy approach to include response matrix data, recovering adaptive-TAP equations and enhancing approximation accuracy.

Findings

Significant improvement in marginal estimation over standard methods in weak coupling regimes.

Enhanced accuracy in frustrated models with short loops.

Framework unifies and generalizes mean-field and Bethe approximations.

Abstract

Approximating marginals of a graphical model is one of the fundamental problems in the theory of networks. In a recent paper a method was shown to construct a variational free energy such that the linear response estimates, and maximum entropy estimates (for beliefs) are in agreement, with implications for direct and inverse Ising problems[1]. In this paper we demonstrate an extension of that method, incorporating new information from the response matrix, and we recover the adaptive-TAP equations as the first order approximation[2]. The method is flexible with respect to applications of the cluster variational method, special cases of this method include Naive Mean Field (NMF) and Bethe. We demonstrate that the new framework improves estimation of marginals by orders of magnitude over standard implementations in the weak coupling limit. Beyond the weakly coupled regime we show there is an improvement in a model where the NMF and Bethe approximations are known to be poor for reasons of frustration and short loops.