Partitioning the Sample Space for a More Precise Shannon Entropy Estimation

TL;DR

This paper introduces a new discrete Shannon entropy estimator that improves accuracy in small data regimes by partitioning the sample space and compensating for unseen outcomes, outperforming classical methods.

Contribution

The paper proposes a novel entropy estimation method leveraging sample space partitioning and missing mass estimation to reduce bias in undersampled data.

Findings

Outperforms classical estimators in undersampled regimes

Performs comparably with state-of-the-art estimators

Effective in small data scenarios

Abstract

Reliable data-driven estimation of Shannon entropy from small data sets, where the number of examples is potentially smaller than the number of possible outcomes, is a critical matter in several applications. In this paper, we introduce a discrete entropy estimator, where we use the decomposability property in combination with estimations of the missing mass and the number of unseen outcomes to compensate for the negative bias induced by them. Experimental results show that the proposed method outperforms some classical estimators in undersampled regimes, and performs comparably with some well-established state-of-the-art estimators.