Maximum a Posteriori Inference for Factor Graphs via Benders' Decomposition

TL;DR

This paper introduces a Benders' decomposition-based method for MAP inference in Bayesian factor graphs, providing convergence guarantees and handling complex constraints, outperforming traditional sampling and variational methods.

Contribution

The paper presents a novel MAP inference algorithm using Benders' decomposition that incorporates complex constraints and offers convergence certificates.

Findings

Achieves higher posterior values than Gibbs sampling and variational Bayes.

Provides convergence certificates for MAP inference.

Handles expressive logical and integer constraints in clustering.

Abstract

Many Bayesian statistical inference problems come down to computing a maximum a-posteriori (MAP) assignment of latent variables. Yet, standard methods for estimating the MAP assignment do not have a finite time guarantee that the algorithm has converged to a fixed point. Previous research has found that MAP inference can be represented in dual form as a linear programming problem with a non-polynomial number of constraints. A Lagrangian relaxation of the dual yields a statistical inference algorithm as a linear programming problem. However, the decision as to which constraints to remove in the relaxation is often heuristic. We present a method for maximum a-posteriori inference in general Bayesian factor models that sequentially adds constraints to the fully relaxed dual problem using Benders' decomposition. Our method enables the incorporation of expressive integer and logical constraints in clustering problems such as must-link, cannot-link, and a minimum number of whole samples allocated to each cluster. Using this approach, we derive MAP estimation algorithms for the Bayesian Gaussian mixture model and latent Dirichlet allocation. Empirical results show that our method produces a higher optimal posterior value compared to Gibbs sampling and variational Bayes methods for standard data sets and provides certificate of convergence.