A Probabilistic Upper Bound on Differential Entropy

TL;DR

This paper introduces a new probabilistic upper bound on the differential entropy of an unknown one-dimensional distribution using only its support and a sample, without requiring a density or additional knowledge.

Contribution

It presents a novel, distribution-free entropy upper bound based on existing bounds on the cumulative distribution function, along with an efficient algorithm for computation.

Findings

Provides a non-trivial probabilistic upper bound on differential entropy

The bound is computed efficiently from samples and known support

Applicable without requiring the distribution to have a density

Abstract

A novel, non-trivial, probabilistic upper bound on the entropy of an unknown one-dimensional distribution, given the support of the distribution and a sample from that distribution, is presented. No knowledge beyond the support of the unknown distribution is required, nor is the distribution required to have a density. Previous distribution-free bounds on the cumulative distribution function of a random variable given a sample of that variable are used to construct the bound. A simple, fast, and intuitive algorithm for computing the entropy bound from a sample is provided.