Kernel Density Matrices for Probabilistic Deep Learning

TL;DR

This paper introduces kernel density matrices, a new probabilistic modeling approach in deep learning that extends quantum density matrices into reproducing kernel Hilbert spaces, enabling flexible, differentiable density estimation and inference.

Contribution

It presents a novel framework for probabilistic deep learning using kernel density matrices, allowing for versatile, differentiable, and reversible density modeling in neural networks.

Findings

Effective joint probability distribution representation

Versatile models for density estimation, inference, and sampling

Successful application to image classification and label proportion learning

Abstract

This paper introduces a novel approach to probabilistic deep learning, kernel density matrices, which provide a simpler yet effective mechanism for representing joint probability distributions of both continuous and discrete random variables. In quantum mechanics, a density matrix is the most general way to describe the state of a quantum system. This work extends the concept of density matrices by allowing them to be defined in a reproducing kernel Hilbert space. This abstraction allows the construction of differentiable models for density estimation, inference, and sampling, and enables their integration into end-to-end deep neural models. In doing so, we provide a versatile representation of marginal and joint probability distributions that allows us to develop a differentiable, compositional, and reversible inference procedure that covers a wide range of machine learning tasks, including density estimation, discriminative learning, and generative modeling. The broad applicability of the framework is illustrated by two examples: an image classification model that can be naturally transformed into a conditional generative model, and a model for learning with label proportions that demonstrates the framework's ability to deal with uncertainty in the training samples. The framework is implemented as a library and is available at: https://github.com/fagonzalezo/kdm.