An algebraic generalization of the entropy and its application to statistics

TL;DR

This paper introduces an algebraic framework for defining a generalized entropy, deriving related scalar products and distances, and connecting these concepts to established statistical measures and inequalities, offering new insights into statistical theory.

Contribution

It presents a novel algebraic generalization of entropy and related measures, unifying various statistical concepts under a common framework.

Findings

Generalized entropy defined algebraically

Derived scalar product and distance measures

Connected to likelihood, divergence, and mutual information

Abstract

We define a general notion of entropy in elementary, algebraic terms. Based on that, weak forms of a scalar product and a distance measure are derived. We give basic properties of these quantities, generalize the Cauchy-Schwarz inequality, and relate our approach to the theory of scoring rules. Many supporting examples illustrate our approach and give new perspectives on established notions, such as the likelihood, the Kullback-Leibler divergence, the uncorrelatedness of random variables, the scalar product itself, the Tichonov regularization and the mutual information.