Tractable Representation Learning with Probabilistic Circuits

TL;DR

This paper introduces autoencoding probabilistic circuits (APCs), a novel method that leverages the tractability of PCs for probabilistic embedding modeling, outperforming existing methods in reconstruction and robustness, and enabling flexible, end-to-end learning.

Contribution

The paper proposes APCs, a new framework that models probabilistic embeddings with PCs, integrating them with neural decoders for end-to-end training and improved handling of missing data.

Findings

APCs outperform existing PC-based autoencoders in reconstruction quality.

APCs generate embeddings that are competitive with neural autoencoders.

APCs exhibit superior robustness in handling missing data.

Abstract

Probabilistic circuits (PCs) are powerful probabilistic models that enable exact and tractable inference, making them highly suitable for probabilistic reasoning and inference tasks. While dominant in neural networks, representation learning with PCs remains underexplored, with prior approaches relying on external neural embeddings or activation-based encodings. To address this gap, we introduce autoencoding probabilistic circuits (APCs), a novel framework leveraging the tractability of PCs to model probabilistic embeddings explicitly. APCs extend PCs by jointly modeling data and embeddings, obtaining embedding representations through tractable probabilistic inference. The PC encoder allows the framework to natively handle arbitrary missing data and is seamlessly integrated with a neural decoder in a hybrid, end-to-end trainable architecture enabled by differentiable sampling. Our empirical evaluation demonstrates that APCs outperform existing PC-based autoencoding methods in reconstruction quality, generate embeddings competitive with, and exhibit superior robustness in handling missing data compared to neural autoencoders. These results highlight APCs as a powerful and flexible representation learning method that exploits the probabilistic inference capabilities of PCs, showing promising directions for robust inference, out-of-distribution detection, and knowledge distillation.