Effective Sparsity: A Unified Framework via Normalized Entropy and the Effective Number of Nonzeros

TL;DR

This paper introduces the effective number of nonzeros (ENZ), a normalized entropy-based regularizer that better captures true sparsity in inverse problems, offering stability, robustness, and improved accuracy over traditional l0 methods.

Contribution

The paper proposes ENZ, a novel entropy-based measure of effective sparsity, providing a continuous, stable alternative to l0 norm with theoretical guarantees and superior practical performance.

Findings

ENZ provides a stable, continuous measure of sparsity.

Theoretical guarantees under RIP for noisy inverse problems.

Numerical results outperform traditional methods in robustness and accuracy.

Abstract

Classical sparsity promoting methods rely on the l0 norm, which treats all nonzero components as equally significant. In practical inverse problems, however, solutions often exhibit many small amplitude components that have little effect on reconstruction but lead to an overestimation of signal complexity. We address this limitation by shifting the paradigm from discrete cardinality to effective sparsity. Our approach introduces the effective number of nonzeros (ENZ), a unified class of normalized entropy-based regularizers, including Shannon and Renyi forms, that quantifies the concentration of significant coefficients. We show that, unlike the classical l0 norm, the ENZ provides a stable and continuous measure of effective sparsity that is insensitive to negligible perturbations. For noisy linear inverse problems, we establish theoretical guarantees under the Restricted Isometry Property (RIP), proving that ENZ based recovery is unique and stable. We also derive a decomposition showing that the ENZ equals the support cardinality times a distributional efficiency term, thereby linking entropy with l0 regularization. Numerical experiments show that this effective sparsity framework outperforms traditional cardinality based methods in robustness and accuracy.