On Bregman Distances and Divergences of Probability Measures

TL;DR

This paper introduces scaled Bregman distances for probability measures, extending classical divergences and applying to various stochastic processes, with implications for information theory, data analysis, and signal processing.

Contribution

It generalizes Bregman distances to include non-uniform event contributions and extends their application to stochastic processes and exponential families.

Findings

Extended Bregman distances to include non-uniform contributions.

Derived explicit formulas for exponential family distributions.

Illustrated applications in stochastic processes and 3D data analysis.

Abstract

The paper introduces scaled Bregman distances of probability distributions which admit non-uniform contributions of observed events. They are introduced in a general form covering not only the distances of discrete and continuous stochastic observations, but also the distances of random processes and signals. It is shown that the scaled Bregman distances extend not only the classical ones studied in the previous literature, but also the information divergence and the related wider class of convex divergences of probability measures. An information processing theorem is established too, but only in the sense of invariance w.r.t. statistically sufficient transformations and not in the sense of universal monotonicity. Pathological situations where coding can increase the classical Bregman distance are illustrated by a concrete example. In addition to the classical areas of application of the Bregman distances and convex divergences such as recognition, classification, learning and evaluation of proximity of various features and signals, the paper mentions a new application in 3D-exploratory data analysis. Explicit expressions for the scaled Bregman distances are obtained in general exponential families, with concrete applications in the binomial, Poisson and Rayleigh families, and in the families of exponential processes such as the Poisson and diffusion processes including the classical examples of the Wiener process and geometric Brownian motion.