Monotonic Convergence in an Information-Theoretic Law of Small Numbers

TL;DR

This paper establishes an entropy increase and convergence result in the discrete setting, linking the law of small numbers with the information-theoretic central limit theorem using the thinning operation and Poisson limits.

Contribution

It extends the analogy between the information-theoretic CLT and the law of small numbers to discrete distributions, proving monotonic entropy properties.

Findings

Monotonic convergence in relative entropy for general discrete distributions

Monotonic increase of Shannon entropy for ultra-log-concave distributions

Extension of the parallel between CLT and law of small numbers

Abstract

An "entropy increasing to the maximum" result analogous to the entropic central limit theorem (Barron 1986; Artstein et al. 2004) is obtained in the discrete setting. This involves the thinning operation and a Poisson limit. Monotonic convergence in relative entropy is established for general discrete distributions, while monotonic increase of Shannon entropy is proved for the special class of ultra-log-concave distributions. Overall we extend the parallel between the information-theoretic central limit theorem and law of small numbers explored by Kontoyiannis et al. (2005) and Harremo\"es et al.\ (2007, 2008). Ingredients in the proofs include convexity, majorization, and stochastic orders.