An Effective Bernstein-type Bound on Shannon Entropy over Countably Infinite Alphabets

TL;DR

This paper establishes a Bernstein-type bound for Shannon entropy differences over countably infinite alphabets, applicable to common distributions with light tails, and provides a method to compute the bound's constants.

Contribution

It introduces a novel Bernstein-type bound for Shannon entropy over infinite alphabets, with an effective method to compute the constants involved.

Findings

The bound applies to distributions with tails lighter than or equal to a power-law.

It covers distributions like Poisson, negative binomial, and power-law.

Provides a practical method to compute the constants in the bound.

Abstract

We prove a Bernstein-type bound for the difference between the average of negative log-likelihoods of independent discrete random variables and the Shannon entropy, both defined on a countably infinite alphabet. The result holds for the class of discrete random variables with tails lighter than or on the same order of a discrete power-law distribution. Most commonly-used discrete distributions such as the Poisson distribution, the negative binomial distribution, and the power-law distribution itself belong to this class. The bound is effective in the sense that we provide a method to compute the constants in it.