Entropy, Optimization and Counting

TL;DR

This paper establishes a theoretical connection between counting discrete objects and computing maximum-entropy distributions, providing conditions for polynomial descriptions and algorithms for approximation.

Contribution

It proves the existence of polynomial-sized descriptions for max-entropy distributions and shows an equivalence between counting problems and max-entropy computation.

Findings

Polynomial descriptions of max-entropy distributions exist under general conditions.

Algorithms for counting can be adapted to compute max-entropy distributions efficiently.

Access to max-entropy algorithms can be used to count discrete objects, establishing an equivalence.

Abstract

In this paper we study the problem of computing max-entropy distributions over a discrete set of objects subject to observed marginals. Interest in such distributions arises due to their applicability in areas such as statistical physics, economics, biology, information theory, machine learning, combinatorics and, more recently, approximation algorithms. A key difficulty in computing max-entropy distributions has been to show that they have polynomially-sized descriptions. We show that such descriptions exist under general conditions. Subsequently, we show how algorithms for (approximately) counting the underlying discrete set can be translated into efficient algorithms to (approximately) compute max-entropy distributions. In the reverse direction, we show how access to algorithms that compute max-entropy distributions can be used to count, which establishes an equivalence between counting and computing max-entropy distributions.