Lossless Compression with Probabilistic Circuits

TL;DR

This paper introduces Probabilistic Circuits, a new class of neural network models that enable efficient lossless data compression with scalable encoding and decoding, outperforming existing neural methods in speed and quality.

Contribution

The paper presents Probabilistic Circuits as a scalable, efficient model for lossless compression, with novel encoding schemes and state-of-the-art results on image datasets.

Findings

PC-based compression runs 5-40 times faster than similar neural methods.

Achieved state-of-the-art results on MNIST dataset.

PCs can be integrated with existing neural models to enhance performance.

Abstract

Despite extensive progress on image generation, common deep generative model architectures are not easily applied to lossless compression. For example, VAEs suffer from a compression cost overhead due to their latent variables. This overhead can only be partially eliminated with elaborate schemes such as bits-back coding, often resulting in poor single-sample compression rates. To overcome such problems, we establish a new class of tractable lossless compression models that permit efficient encoding and decoding: Probabilistic Circuits (PCs). These are a class of neural networks involving $|p|$ computational units that support efficient marginalization over arbitrary subsets of the $D$ feature dimensions, enabling efficient arithmetic coding. We derive efficient encoding and decoding schemes that both have time complexity $\mathcal{O} (\log(D) \cdot |p|)$, where a naive scheme would have linear costs in $D$ and $|p|$, making the approach highly scalable. Empirically, our PC-based (de)compression algorithm runs 5-40 times faster than neural compression algorithms that achieve similar bitrates. By scaling up the traditional PC structure learning pipeline, we achieve state-of-the-art results on image datasets such as MNIST. Furthermore, PCs can be naturally integrated with existing neural compression algorithms to improve the performance of these base models on natural image datasets. Our results highlight the potential impact that non-standard learning architectures may have on neural data compression.