Generalising maximum mean discrepancy: kernelised functional Bregman divergences

TL;DR

This paper extends the concept of Bregman divergences to Hilbert spaces using kernel methods, enabling new applications in machine learning such as clustering and generative modeling.

Contribution

It develops a systematic framework for functional Bregman divergences in Hilbert spaces, integrating kernel mean embeddings for easier estimation.

Findings

Framework for kernelised functional Bregman divergences

Applications demonstrated in clustering and generative modeling

Comparison with other Bregman divergence types

Abstract

Bregman divergences play a pivotal role in statistics, machine learning and computational information geometry. Particularly in the context of machine learning, they are central to clustering, exponential families, parameter estimation and optimisation, among other things. Despite this, the full toolkit of Hilbert spaces and in particular reproducing kernel Hilbert spaces have not been systematically developed and applied to functional Bregman divergences, where points are functions rather than finite-dimensional parameter vectors. While other types of functional Bregman divergences have been studied, these are typically in a Banach space rather than more directly aligned with kernel methods and Hilbert-space geometry commonly used in machine learning. We consider functional Bregman divergences on a Hilbert space, where the self-dual pairing and Riesz representer afford us particularly convenient calculus. Further specialising Bregman generators as a composition involving a kernel mean embedding makes such divergences easy to estimate. We discuss applications in clustering, universal estimation, robust estimation and generative modelling, and contrast our approach with other types of Bregman divergences.