Codivergences and information matrices

TL;DR

This paper introduces the concept of codivergence to measure similarity between probability measures relative to a reference, exploring covariance and correlation types, and establishing properties like the data-processing inequality.

Contribution

It proposes the novel concept of codivergence, develops divergence matrices, and analyzes specific correlation-type codivergences such as the chi-squared and Hellinger.

Findings

Explicit formulas for common parametric families

Introduction of divergence matrices as Gram matrix analogues

Chi-squared divergence matrix satisfies data-processing inequality

Abstract

We propose a new concept of codivergence, which quantifies the similarity between two probability measures $P_1, P_2$ relative to a reference probability measure $P_0$. In the neighborhood of the reference measure $P_0$, a codivergence behaves like an inner product between the measures $P_1 - P_0$ and $P_2 - P_0$. Codivergences of covariance-type and correlation-type are introduced and studied with a focus on two specific correlation-type codivergences, the $\chi^2$-codivergence and the Hellinger codivergence. We derive explicit expressions for several common parametric families of probability distributions. For a codivergence, we introduce moreover the divergence matrix as an analogue of the Gram matrix. It is shown that the $\chi^2$-divergence matrix satisfies a data-processing inequality.