Some Properties of R\'{e}nyi Entropy over Countably Infinite Alphabets

TL;DR

This paper investigates the properties of Rényi entropy for probability distributions over countably infinite sets, highlighting differences from finite cases and addressing continuity and convergence issues.

Contribution

It characterizes the behavior of Rényi entropy in infinite alphabets, revealing unique properties and convergence phenomena not present in finite cases.

Findings

Rényi entropy exhibits distinct behavior in infinite alphabets.

Existence of distribution sequences with specific convergence properties.

Differences in limits of Rényi entropy as alpha approaches 1.

Abstract

In this paper we study certain properties of R\'{e}nyi entropy functionals $H_\alpha(\mathcal{P})$ on the space of probability distributions over $\mathbb{Z}_+$. Primarily, continuity and convergence issues are addressed. Some properties shown parallel those known in the finite alphabet case, while others illustrate a quite different behaviour of R\'enyi entropy in the infinite case. In particular, it is shown that, for any distribution $\mathcal P$ and any $r\in[0,\infty]$, there exists a sequence of distributions $\mathcal{P}_n$ converging to $\mathcal{P}$ with respect to the total variation distance, such that $\lim_{n\to\infty}\lim_{\alpha\to{1+}} H_\alpha(\mathcal{P}_n) = \lim_{\alpha\to{1+}}\lim_{n\to\infty} H_\alpha(\mathcal{P}_n) + r$.