Approximate maximum entropy principles via Goemans-Williamson with applications to provable variational methods

TL;DR

This paper introduces efficient methods for approximate maximum entropy distributions based on pairwise moments, enabling provable variational algorithms for Ising models with broad temperature applicability.

Contribution

It develops computationally feasible approximate maximum entropy principles using Goemans-Williamson techniques, with novel applications to variational methods for partition function estimation.

Findings

Provides approximation guarantees for log-partition functions across all temperatures.

Designs distributions matching pairwise moments with entropy close to true maximum.

Enables provable variational algorithms for Ising models without potential assumptions.

Abstract

The well known maximum-entropy principle due to Jaynes, which states that given mean parameters, the maximum entropy distribution matching them is in an exponential family, has been very popular in machine learning due to its "Occam's razor" interpretation. Unfortunately, calculating the potentials in the maximum-entropy distribution is intractable \cite{bresler2014hardness}. We provide computationally efficient versions of this principle when the mean parameters are pairwise moments: we design distributions that approximately match given pairwise moments, while having entropy which is comparable to the maximum entropy distribution matching those moments. We additionally provide surprising applications of the approximate maximum entropy principle to designing provable variational methods for partition function calculations for Ising models without any assumptions on the potentials of the model. More precisely, we show that in every temperature, we can get approximation guarantees for the log-partition function comparable to those in the low-temperature limit, which is the setting of optimization of quadratic forms over the hypercube. \cite{alon2006approximating}