Differential Shannon and R\'enyi entropies revisited

TL;DR

This paper critically examines the limitations of differential Shannon and Renyi entropies, proposing modified versions that preserve key properties like positivity and compatibility with discrete cases, supported by analysis on common distributions.

Contribution

It introduces new modified entropy measures that address fundamental issues of classical differential entropies, ensuring positivity and discrete compatibility.

Findings

Modified entropies retain positivity and compatibility

Analysis of entropies for normal and exponential distributions

Proposed functionals improve theoretical consistency

Abstract

Shannon entropy for discrete distributions is a fundamental and widely used concept, but its continuous analogue, known as differential entropy, lacks essential properties such as positivity and compatibility with the discrete case. In this paper, we analyze this incompatibility in detail and illustrate it through examples. To overcome these limitations, we propose modified versions of Shannon and R\'enyi entropy that retain key properties, including positivity, while remaining close to the classical forms. We also define compatible discrete functionals and study the behavior of the proposed entropies for the normal and exponential distributions.