Mathematics of Sparsity and Entropy: Axioms, Core Functions and Sparse Recovery

TL;DR

This paper develops a unified mathematical framework with axioms to characterize sparsity and entropy, deriving core functions that generalize existing measures and support applications like compressed sensing.

Contribution

It introduces a joint axiomatic formalism for sparsity and entropy, deriving core functions that unify and extend existing measures, with applications to signal processing.

Findings

Core sparsity supports -minimization in compressed sensing.

Core functions generalize key measures like Gini index and Re9nyi entropy.

The formalism provides a unified understanding of sparsity and entropy.

Abstract

Sparsity and entropy are pillar notions of modern theories in signal processing and information theory. However, there is no clear consensus among scientists on the characterization of these notions. Previous efforts have contributed to understand individually sparsity or entropy from specific research interests. This paper proposes a mathematical formalism, a joint axiomatic characterization, which contributes to comprehend (the beauty of) sparsity and entropy. The paper gathers and introduces inherent and first principles criteria as axioms and attributes that jointly characterize sparsity and entropy. The proposed set of axioms is constructive and allows to derive simple or \emph{core functions} and further generalizations. Core sparsity generalizes the Hoyer measure, Gini index and $pq$-means. Core entropy generalizes the R\'{e}nyi entropy and Tsallis entropy, both of which generalize Shannon entropy. Finally, core functions are successfully applied to compressed sensing and to minimum entropy given sample moments. More importantly, the (simplest) core sparsity adds theoretical support to the $\ell_1$-minimization approach in compressed sensing.