Similarity-Sensitive Entropy: Induced Kernels and Data-Processing Inequalities

TL;DR

This paper introduces a similarity-sensitive entropy functional that generalizes existing concepts, explores its properties under coarse-graining, and establishes data-processing inequalities in a measure-theoretic framework.

Contribution

It develops a measure-theoretic framework for similarity-sensitive entropy, proves coarse-graining and data-processing inequalities, and introduces conditional entropy and mutual information in this setting.

Findings

H_K satisfies coarse-graining inequalities.

H_K obeys data-processing inequalities for Markov kernels.

Introduces conditional entropy and mutual information compatible with similarity structures.

Abstract

We study an entropy functional $H_K$ that is sensitive to a prescribed similarity structure on a state space. For finite spaces, $H_K$ coincides with the order-1 similarity-sensitive entropy of Leinster and Cobbold. We work in the general measure-theoretic setting of kernelled probability spaces $(\Omega,\mu,K)$ introduced by Leinster and Roff, and develop basic structural properties of $H_K$. Our main results concern the behavior of $H_K$ under coarse-graining. For a measurable map $f:\Omega\to Y$ and input law $\mu$, we define a law-induced kernel on $Y$ whose pullback minimally dominates $K$, and show that it yields a coarse-graining inequality and a data-processing inequality for $H_K$, for both deterministic maps and general Markov kernels. We also introduce conditional similarity-sensitive entropy and an associated mutual information, and compare their behavior to the classical Shannon case.