Approximate Inference via Weighted Rademacher Complexity

TL;DR

This paper introduces a novel technique to estimate the size of weighted sets using a generalized Rademacher complexity, enabling tighter bounds for problems like partition functions and model counting.

Contribution

It extends Rademacher complexity to weighted sets and provides a practical estimation method via optimization, improving bounds in complex probabilistic models.

Findings

Tighter bounds on weighted set sizes than existing methods

Effective estimation of partition functions in Ising models

Improved propositional model counting accuracy

Abstract

Rademacher complexity is often used to characterize the learnability of a hypothesis class and is known to be related to the class size. We leverage this observation and introduce a new technique for estimating the size of an arbitrary weighted set, defined as the sum of weights of all elements in the set. Our technique provides upper and lower bounds on a novel generalization of Rademacher complexity to the weighted setting in terms of the weighted set size. This generalizes Massart's Lemma, a known upper bound on the Rademacher complexity in terms of the unweighted set size. We show that the weighted Rademacher complexity can be estimated by solving a randomly perturbed optimization problem, allowing us to derive high-probability bounds on the size of any weighted set. We apply our method to the problems of calculating the partition function of an Ising model and computing propositional model counts (#SAT). Our experiments demonstrate that we can produce tighter bounds than competing methods in both the weighted and unweighted settings.