Measuring Dependencies of Order Statistics: An Information Theoretic Perspective

TL;DR

This paper investigates the dependencies among order statistics using information theory, establishing distribution-free properties for continuous distributions and analyzing the influence of distribution on mutual information in discrete cases.

Contribution

It introduces distribution-free measures of dependence among order statistics for continuous distributions and explores how these dependencies vary in discrete distributions.

Findings

Distribution-free mutual information properties for continuous distributions.

Decoupling rates of order statistics depend on their positions.

Mutual information in discrete cases depends on the underlying distribution.

Abstract

Consider a random sample $X_1 , X_2 , ..., X_n$ drawn independently and identically distributed from some known sampling distribution $P_X$. Let $X_{(1)} \le X_{(2)} \le ... \le X_{(n)}$ represent the order statistics of the sample. The first part of the paper focuses on distributions with an invertible cumulative distribution function. Under this assumption, a distribution-free property is established, which shows that the $f$-divergence between the joint distribution of order statistics and the product distribution of order statistics does not depend on the original sampling distribution $P_X$. Moreover, it is shown that the mutual information between two subsets of order statistics also satisfies a distribution-free property; that is, it does not depend on $P_X$. Furthermore, the decoupling rates between $X_{(r)}$ and $X_{(m)}$ (i.e., rates at which the mutual information approaches zero) are characterized for various choices of $(r,m)$. The second part of the paper considers a family of discrete distributions, which does not satisfy the assumptions in the first part of the paper. In comparison to the results of the first part, it is shown that in the discrete setting, the mutual information between order statistics does depend on the sampling distribution $P_X$. Nonetheless, it is shown that the results of the first part can still be used as upper bounds on the decoupling rates.