Deep Divergence Learning

TL;DR

This paper introduces deep Bregman divergences, a neural network-based framework that unifies and extends existing deep metric learning and divergence measures, demonstrating superior performance on benchmarks and enabling new applications.

Contribution

It proposes a novel deep Bregman divergence framework that unifies various metric learning approaches and introduces new applications in clustering and data generation.

Findings

Outperforms existing deep metric learning methods on benchmark datasets

Unifies multiple divergence measures within a single neural network framework

Enables novel semi-supervised and unsupervised learning applications

Abstract

Classical linear metric learning methods have recently been extended along two distinct lines: deep metric learning methods for learning embeddings of the data using neural networks, and Bregman divergence learning approaches for extending learning Euclidean distances to more general divergence measures such as divergences over distributions. In this paper, we introduce deep Bregman divergences, which are based on learning and parameterizing functional Bregman divergences using neural networks, and which unify and extend these existing lines of work. We show in particular how deep metric learning formulations, kernel metric learning, Mahalanobis metric learning, and moment-matching functions for comparing distributions arise as special cases of these divergences in the symmetric setting. We then describe a deep learning framework for learning general functional Bregman divergences, and show in experiments that this method yields superior performance on benchmark datasets as compared to existing deep metric learning approaches. We also discuss novel applications, including a semi-supervised distributional clustering problem, and a new loss function for unsupervised data generation.