Information Measures for Entropy and Symmetry

TL;DR

This paper introduces new information measures that generalize entropy as a subspace volume, clarifying its relationship with symmetry and uniformity in topological groups, and addressing issues with negative entropies in continuous settings.

Contribution

It proposes sup-normalization and new information measures to extend entropy's interpretation beyond discrete cases, linking entropy with symmetry in topological groups.

Findings

New measures avoid negative entropy in continuous cases

Entropy interpreted as subspace volume in topological groups

Clarifies the link between entropy, symmetry, and uniformity

Abstract

Entropy and information can be considered dual: entropy is a measure of the subspace defined by the information constraining the given ambient space. Negative entropies, arising in na\"ive extensions of the definition of entropy from discrete to continuous settings, are byproducts of the use of probabilities, which only work in the discrete case by a fortunate coincidence. We introduce the notions of sup-normalization and information measures, which allow for the appropriate generalization of the definition of entropy that keeps with the interpretation of entropy as a subspace volume. Applying this in the context of topological groups and Haar measures, we elucidate the relationship between entropy, symmetry, and uniformity.