On Convergence Properties of Shannon Entropy

TL;DR

This paper investigates the convergence properties of Shannon Entropy, revealing that weak convergence alone does not ensure entropy convergence and providing conditions for such convergence in both differential and discrete cases.

Contribution

It offers new conditions for differential entropy convergence considering support types and links convergence to the Kullback-Leibler divergence, extending understanding to discrete measures.

Findings

Weak convergence does not imply differential entropy convergence.

Convergence in variation guarantees entropy convergence under boundedness.

Results extend to discrete measures with infinite support.

Abstract

Convergence properties of Shannon Entropy are studied. In the differential setting, it is shown that weak convergence of probability measures, or convergence in distribution, is not enough for convergence of the associated differential entropies. A general result for the desired differential entropy convergence is provided, taking into account both compactly and uncompactly supported densities. Convergence of differential entropy is also characterized in terms of the Kullback-Liebler discriminant for densities with fairly general supports, and it is shown that convergence in variation of probability measures guarantees such convergence under an appropriate boundedness condition on the densities involved. Results for the discrete setting are also provided, allowing for infinitely supported probability measures, by taking advantage of the equivalence between weak convergence and convergence in variation in this setting.