Robust and Fast Measure of Information via Low-rank Representation

TL;DR

This paper introduces a robust, computationally efficient low-rank variant of matrix-based Rényi's entropy that effectively quantifies information from data while being resistant to noise, suitable for large-scale applications.

Contribution

The authors propose a novel low-rank Rényi's entropy measure that enhances robustness and reduces computational complexity using random projections and Lanczos iteration.

Findings

Outperforms original Rényi's entropy in noisy data scenarios

Achieves significant computational speed-up in large-scale data

Demonstrates superior accuracy and efficiency in experiments

Abstract

The matrix-based R\'enyi's entropy allows us to directly quantify information measures from given data, without explicit estimation of the underlying probability distribution. This intriguing property makes it widely applied in statistical inference and machine learning tasks. However, this information theoretical quantity is not robust against noise in the data, and is computationally prohibitive in large-scale applications. To address these issues, we propose a novel measure of information, termed low-rank matrix-based R\'enyi's entropy, based on low-rank representations of infinitely divisible kernel matrices. The proposed entropy functional inherits the specialty of of the original definition to directly quantify information from data, but enjoys additional advantages including robustness and effective calculation. Specifically, our low-rank variant is more sensitive to informative perturbations induced by changes in underlying distributions, while being insensitive to uninformative ones caused by noises. Moreover, low-rank R\'enyi's entropy can be efficiently approximated by random projection and Lanczos iteration techniques, reducing the overall complexity from $\mathcal{O}(n^3)$ to $\mathcal{O}(n^2 s)$ or even $\mathcal{O}(ns^2)$, where $n$ is the number of data samples and $s \ll n$. We conduct large-scale experiments to evaluate the effectiveness of this new information measure, demonstrating superior results compared to matrix-based R\'enyi's entropy in terms of both performance and computational efficiency.