Sum-Product-Attention Networks: Leveraging Self-Attention in Probabilistic Circuits

TL;DR

Sum-Product-Attention Networks (SPAN) integrate self-attention mechanisms with probabilistic circuits, specifically sum-product networks, to enhance generative modeling and inference capabilities, outperforming existing models on benchmarks.

Contribution

The paper introduces SPAN, a novel model combining Transformers with probabilistic circuits, enabling selective attention within sum-product networks for improved modeling.

Findings

SPAN outperforms state-of-the-art probabilistic models on benchmarks.

SPAN is an efficient generative image model.

Self-attention in SPAN improves the modeling of independent assumptions.

Abstract

Probabilistic circuits (PCs) have become the de-facto standard for learning and inference in probabilistic modeling. We introduce Sum-Product-Attention Networks (SPAN), a new generative model that integrates probabilistic circuits with Transformers. SPAN uses self-attention to select the most relevant parts of a probabilistic circuit, here sum-product networks, to improve the modeling capability of the underlying sum-product network. We show that while modeling, SPAN focuses on a specific set of independent assumptions in every product layer of the sum-product network. Our empirical evaluations show that SPAN outperforms state-of-the-art probabilistic generative models on various benchmark data sets as well is an efficient generative image model.