Expressing entropy and cross-entropy in expansions of common meadows

TL;DR

This paper introduces a novel algebraic framework using common meadows to express entropy and cross-entropy, enabling definitions without case distinctions or partiality, thus simplifying the mathematical treatment of information measures.

Contribution

It develops a new algebraic approach based on common meadows to define entropy and cross-entropy uniformly for all arguments, avoiding case distinctions.

Findings

Algebras over common meadows allow simple algebraic formulas for entropy.

Entropy can be expressed without case distinctions using these algebras.

The approach simplifies the mathematical treatment of information measures.

Abstract

A common meadow is an enrichment of a field with a partial division operation that is made total by assuming that division by zero takes the a default value, a special element $\bot$ adjoined to the field. To a common meadow of real numbers we add a binary logarithm $\log_2(-)$, which we also assume to be total with $\log_2(p) = \bot$ for $p \leq 0$. With these and other auxiliary operations, such as a sign function, we form algebras over which entropy and cross entropy can be defined for probability mass functions on a finite sample space by algebraic formulae that are simple terms built from the operations of the algebras and without case distinctions or conventions to avoid partiality. The discuss the advantages of algebras based on common meadows, whose theory is established, and alternate methods to define entropy and other information measures completely for all arguments using single terms.