Information, Divergence and Risk for Binary Experiments

TL;DR

This paper unifies various divergence measures and loss functions within a common framework related to binary classification, leading to new bounds, inequalities, and insights into existing algorithms.

Contribution

It introduces a systematic integral and variational approach that unifies divergences, loss bounds, and information measures, revealing new relationships and derivations in binary learning.

Findings

Derived tight surrogate loss bounds

Established generalized Pinsker inequalities

Provided new derivations of SVMs and links to MMD

Abstract

We unify f-divergences, Bregman divergences, surrogate loss bounds (regret bounds), proper scoring rules, matching losses, cost curves, ROC-curves and information. We do this by systematically studying integral and variational representations of these objects and in so doing identify their primitives which all are related to cost-sensitive binary classification. As well as clarifying relationships between generative and discriminative views of learning, the new machinery leads to tight and more general surrogate loss bounds and generalised Pinsker inequalities relating f-divergences to variational divergence. The new viewpoint illuminates existing algorithms: it provides a new derivation of Support Vector Machines in terms of divergences and relates Maximum Mean Discrepancy to Fisher Linear Discriminants. It also suggests new techniques for estimating f-divergences.