Meta-probabilistic Modeling

TL;DR

This paper introduces meta-probabilistic modeling (MPM), a hierarchical approach for learning generative models across related datasets, enabling scalable inference and meaningful latent representations.

Contribution

It proposes a novel hierarchical framework with a VAE-inspired surrogate objective and bi-level optimization for scalable learning of shared and dataset-specific structures.

Findings

MPM effectively adapts generative models to data.

It recovers meaningful latent representations.

Demonstrated on object-centric and text modeling tasks.

Abstract

Probabilistic graphical models (PGMs) are widely used to discover latent structure in data, but their success hinges on selecting an appropriate model design. In practice, model specification is difficult and often requires iterative trial-and-error. This challenge arises because classical PGMs typically operate on individual datasets. In this work, we consider settings involving collections of related datasets and propose meta-probabilistic modeling (MPM) to learn the generative model structure itself. MPM uses a hierarchical formulation in which global components encode shared patterns across datasets, while local parameters capture dataset-specific latent structure. For scalable learning and inference, we derive a tractable VAE-inspired surrogate objective together with a bi-level optimization algorithm. Our methodology supports a broad class of expressive probabilistic models and has connections to existing architectures, such as Slot Attention. Experiments on object-centric representation learning and sequential text modeling demonstrate that MPM effectively adapts generative models to data while recovering meaningful latent representations.