The Entropy of Floating-Point Numbers

TL;DR

This paper introduces an analytic approximation for the entropy of floating-point numbers, providing bounds, insights into scaling invariance, and closed-form expressions for common distributions.

Contribution

It presents a novel approximation linking floating-point quantization entropy to a new quantity, with bounds and exact results for various distributions.

Findings

Approximate entropy remains stable under scaling.

Closed-form expressions match well with exact results.

Bounds on the approximation error are established.

Abstract

Here we present an analytic approximation for the entropy of floating-point numbers, along with bounds on the error of this approximation. It is well-known that the differential entropy is tightly linked to the discrete entropy of a uniformly quantized random variable. Our approximation uncovers a different quantity that provides this link for floating-point quantization. Additionally, we prove that the entropy of a floating-point quantized random variable is approximately unchanged under scaling. Closed-form expressions for the floating-point entropy of common distributions are provided and compared to exact results.