Properties of the Shannon, R\'{e}nyi and other entropies: dependence in parameters, robustness in distributions and extremes

TL;DR

This paper analyzes various entropy measures across different probability distributions, examining their parameter dependence, robustness, and extreme values, with implications for understanding distribution characteristics and entropy behavior.

Contribution

It provides a comprehensive analysis of multiple entropy measures for various distributions, highlighting their parameter dependence and identifying extreme entropy values.

Findings

Entropy measures depend on distribution parameters

Shannon entropy converges for certain distributions

Extreme Shannon entropy values are identified for Gaussian vectors

Abstract

We calculate and analyze various entropy measures and their properties for selected probability distributions. The entropies considered include Shannon, R\'enyi, generalized R\'enyi, Tsallis, Sharma-Mittal, and modified Shannon entropy, along with the Kullback-Leibler divergence. These measures are examined for several distributions, including gamma, chi-squared, exponential, Laplace, and log-normal distributions. We investigate the dependence of the entropy on the parameters of the respective distribution. We also study the convergence of Shannon entropy for certain probability distributions. Furthermore, we identify the extreme values of Shannon entropy for Gaussian vectors.