Complete Diagrammatic Axiomatisations of Relative Entropy

TL;DR

This paper provides a categorical framework with complete axiomatizations for relative entropy measures, including Kullback-Leibler and Renyi divergences, using string diagrams and monoidal algebra.

Contribution

It introduces a novel categorical approach to relative entropy, offering complete axiomatisations for divergence measures within a graphical, monoidal algebra framework.

Findings

Complete axiomatizations of Kullback-Leibler divergence.

Extension to Renyi divergences of arbitrary order.

Framework using string diagrams and quantitative monoidal algebra.

Abstract

Relative entropy is a fundamental class of distances between probability distributions, with widespread applications in probability theory, statistics, and machine learning. In this work, we study relative entropy from a categorical perspective, viewing it as a quantitative enrichment of categories of stochastic matrices. We consider two natural monoidal structures on stochastic matrices, given by the Kronecker product and the direct sum. Our main results are complete axiomatisations of Kullback-Leibler divergence and, more generally, of R\'enyi divergences of arbitrary order, for each such structure. Our axiomatic theories are formulated within the framework of quantitative monoidal algebra, using a graphical language of string diagrams enriched with quantitative equations.