A Unifying Framework for Some Directed Distances in Statistics

TL;DR

This paper introduces a comprehensive framework unifying various directed distances and divergences in statistics, encompassing density-based, distribution-function-based, and quantile-based measures, with new dependence concepts and variational representations.

Contribution

It provides a unifying theoretical framework for multiple divergence measures, including novel procedures and dependence concepts beyond mutual information.

Findings

Unified framework covers density and distribution-based divergences

Derives new dependence measures as alternatives to mutual information

Includes variational representations of divergence measures

Abstract

Density-based directed distances -- particularly known as divergences -- between probability distributions are widely used in statistics as well as in the adjacent research fields of information theory, artificial intelligence and machine learning. Prominent examples are the Kullback-Leibler information distance (relative entropy) which e.g. is closely connected to the omnipresent maximum likelihood estimation method, and Pearson's chisquare-distance which e.g. is used for the celebrated chisquare goodness-of-fit test. Another line of statistical inference is built upon distribution-function-based divergences such as e.g. the prominent (weighted versions of) Cramer-von Mises test statistics respectively Anderson-Darling test statistics which are frequently applied for goodness-of-fit investigations; some more recent methods deal with (other kinds of) cumulative paired divergences and closely related concepts. In this paper, we provide a general framework which covers in particular both the above-mentioned density-based and distribution-function-based divergence approaches; the dissimilarity of quantiles respectively of other statistical functionals will be included as well. From this framework, we structurally extract numerous classical and also state-of-the-art (including new) procedures. Furthermore, we deduce new concepts of dependence between random variables, as alternatives to the celebrated mutual information. Some variational representations are discussed, too.