Fast, close, non-singular and property-preserving approximations of entropic measures

TL;DR

This paper introduces Fast Entropic Approximations (FEA), non-singular rational methods for efficiently computing entropic measures like Shannon entropy and KL divergence, significantly improving speed and robustness in various applications.

Contribution

The paper presents novel non-singular rational approximations of entropic measures that preserve key properties and outperform existing methods in speed and accuracy.

Findings

FEA achieves around 10^-3 mean absolute error, 10-20 times better than state-of-the-art approximations.

FEA enables up to 2 times faster Shannon entropy computation and 37 times faster KL divergence calculation.

Using FEA in machine learning accelerates feature extraction by three orders of magnitude, improving both speed and quality.

Abstract

Entropic measures like Shannon entropy (SE), its quantum mechanical analogue von Neumann entropy, and Kullback-Leibler divergence (KL) are key components in many tools used in physics, information theory, machine learning (ML) and quantum computing. Besides of the significant amounts of SE and KL computations required in these fields, the singularity of their gradients near zero is one of the central mathematical reason inducing the high cost, frequently low robustness and slow convergence of computational tools that rely on these concepts. Here we propose the Fast Entropic Approximations (FEA) - non-singular rational approximations of SE and symmetrized KL, that preserve their main mathematical properties and achieve a mean absolute errors of around $10^-3$ ($10-20$ times better than comparable state-of-the-art computational approximations). We show that FEA allows up to around 2 times faster computation of SE and up to 37 times faster computation of symmetrized KL: it requires only $5$ to $7$ elementary computational operations, as compared to the tens of elementary operations behind SE and KL evaluations based on approximate logarithm schemes with table look-ups, bitshifts, or series approximations. On a set of common benchmarks for the feature selection problem in machine learning, we show that the combined effect of fewer elementary operations, low approximation error, preservation of main mathematical properties, and non-singular gradients allows much faster training of significantly-better models. We demonstrate that FEA enables ML feature extraction that is three orders of magnitude faster, and better in quality then the very popular LASSO feature extraction.