Towards Representation Learning with Tractable Probabilistic Models

TL;DR

This paper explores how tractable probabilistic models like Sum-Product Networks and Mixture of Trees can be used to generate meaningful data embeddings through polynomial-time inference, enhancing unsupervised representation learning.

Contribution

It proposes a novel approach to extract features from black box models using tractable inference, with experimental validation on image datasets.

Findings

Sum-Product Networks effectively generate embeddings for images.

Mixture of Trees also serve as useful feature extractors.

Tractable models enable efficient and general feature extraction methods.

Abstract

Probabilistic models learned as density estimators can be exploited in representation learning beside being toolboxes used to answer inference queries only. However, how to extract useful representations highly depends on the particular model involved. We argue that tractable inference, i.e. inference that can be computed in polynomial time, can enable general schemes to extract features from black box models. We plan to investigate how Tractable Probabilistic Models (TPMs) can be exploited to generate embeddings by random query evaluations. We devise two experimental designs to assess and compare different TPMs as feature extractors in an unsupervised representation learning framework. We show some experimental results on standard image datasets by applying such a method to Sum-Product Networks and Mixture of Trees as tractable models generating embeddings.