Unified Perspective on Probability Divergence via Maximum Likelihood Density Ratio Estimation: Bridging KL-Divergence and Integral Probability Metrics

TL;DR

This paper unifies the understanding of KL-divergence and IPMs through maximum likelihood density ratio estimation, introducing Density Ratio Metrics that interpolate between these divergences and applying them to generative modeling.

Contribution

It provides a unified framework linking KL-divergence and IPMs via maximum likelihood density ratio estimation and introduces Density Ratio Metrics as a new class of divergences.

Findings

Unified representation of KL-divergence and IPMs as maximal likelihoods.

Proposed Density Ratio Metrics that interpolate between divergences.

Validated effectiveness of methods in experiments.

Abstract

This paper provides a unified perspective for the Kullback-Leibler (KL)-divergence and the integral probability metrics (IPMs) from the perspective of maximum likelihood density-ratio estimation (DRE). Both the KL-divergence and the IPMs are widely used in various fields in applications such as generative modeling. However, a unified understanding of these concepts has still been unexplored. In this paper, we show that the KL-divergence and the IPMs can be represented as maximal likelihoods differing only by sampling schemes, and use this result to derive a unified form of the IPMs and a relaxed estimation method. To develop the estimation problem, we construct an unconstrained maximum likelihood estimator to perform DRE with a stratified sampling scheme. We further propose a novel class of probability divergences, called the Density Ratio Metrics (DRMs), that interpolates the KL-divergence and the IPMs. In addition to these findings, we also introduce some applications of the DRMs, such as DRE and generative adversarial networks. In experiments, we validate the effectiveness of our proposed methods.