On the Hardness of Approximating Distributions with Tractable Probabilistic Models

TL;DR

This paper investigates the complexity of approximating probability distributions with tractable probabilistic models, revealing fundamental hardness results and size gaps that impact the balance between expressivity and inference efficiency.

Contribution

It introduces the study of approximate distribution representation with probabilistic circuits under $f$-divergences, highlighting NP-hardness and size gaps in model classes.

Findings

Approximating distributions with bounded $f$-divergence is NP-hard for models with tractable marginals.

Exponential size gap exists between decomposable PCs and decomposable deterministic PCs.

Approximate representations can avoid exponential blow-up under certain conditions.

Abstract

A fundamental challenge in probabilistic modeling is to balance expressivity and inference efficiency. Tractable probabilistic models (TPMs) aim to directly address this tradeoff by imposing constraints that guarantee efficient inference of certain queries while maintaining expressivity. In particular, probabilistic circuits (PCs) provide a unifying framework for many TPMs, by characterizing families of models as circuits satisfying different structural properties. Because the complexity of inference on PCs is a function of the circuit size, understanding the size requirements of different families of PCs is fundamental in mapping the trade-off between tractability and expressive efficiency. However, the study of expressive efficiency of circuits are often concerned with exact representations, which may not align with model learning, where we look to approximate the underlying data distribution closely by some distance measure. Moreover, due to hardness of inference tasks, exactly representing distributions while supporting tractable inference often incurs exponential size blow-ups. In this paper, we consider a natural, yet so far underexplored, question: can we avoid such size blow-up by allowing for some small approximation error? We study approximating distributions with probabilistic circuits with guarantees based on $f$-divergences, and analyze which inference queries remain well-approximated under this framework. We show that approximating an arbitrary distribution with bounded $f$-divergence is $\mathsf{NP}$-hard for any model that can tractably compute marginals. In addition, we prove an exponential size gap for approximation between the class of decomposable PCs and that of decomposable and deterministic PCs.