Sum-of-Squares Relaxations for Information Theory and Variational Inference

TL;DR

This paper develops sum-of-squares based convex relaxations for computing generalized divergences, enabling efficient approximation algorithms for problems like estimation, integration, and variational inference in probabilistic models.

Contribution

It introduces a novel sum-of-squares relaxation framework for f-divergences, providing polynomial-time computable approximations for complex information-theoretic and inference problems.

Findings

Sum-of-squares relaxations are effective for approximating f-divergences.

The proposed methods are computationally efficient and polynomial-time.

Illustrations demonstrate applicability to multivariate trigonometric polynomials and Boolean functions.

Abstract

We consider extensions of the Shannon relative entropy, referred to as $f$-divergences.Three classical related computational problems are typically associated with these divergences: (a) estimation from moments, (b) computing normalizing integrals, and (c) variational inference in probabilistic models. These problems are related to one another through convex duality, and for all them, there are many applications throughout data science, and we aim for computationally tractable approximation algorithms that preserve properties of the original problem such as potential convexity or monotonicity. In order to achieve this, we derive a sequence of convex relaxations for computing these divergences from non-centered covariance matrices associated with a given feature vector: starting from the typically non-tractable optimal lower-bound, we consider an additional relaxation based on ``sums-of-squares'', which is is now computable in polynomial time as a semidefinite program. We also provide computationally more efficient relaxations based on spectral information divergences from quantum information theory. For all of the tasks above, beyond proposing new relaxations, we derive tractable convex optimization algorithms, and we present illustrations on multivariate trigonometric polynomials and functions on the Boolean hypercube.