Hutchinson's Estimator is Bad at Kronecker-Trace-Estimation

TL;DR

This paper critically examines Hutchinson's Estimator for Kronecker-structured matrices, revealing its limitations and proposing bounds on its efficiency, with potential improvements when using complex vectors or specialized sampling methods.

Contribution

The paper provides tight bounds on the number of matrix-vector products needed for Hutchinson's Estimator in Kronecker settings, and analyzes its variance and limitations.

Findings

Real vector approach requires exponential samples in the worst case.

Complex vectors significantly reduce the sample complexity.

Structured sampling with low-dimensional vectors improves efficiency.

Abstract

We study the problem of estimating the trace of a matrix $\mathbf{A}$ that can only be accessed through Kronecker-matrix-vector products. That is, for any Kronecker-structured vector $\mathrm{x} = \otimes_{i=1}^k \mathrm{x}_i$, we can compute $\mathbf{A}\mathrm{x}$. We focus on the natural generalization of Hutchinson's Estimator to this setting, proving tight rates for the number of matrix-vector products this estimator needs to find a $(1\pm\varepsilon)$ approximation to the trace of $\mathbf{A}$. We find an exact equation for the variance of the estimator when using a Kronecker of Gaussian vectors, revealing an intimate relationship between Hutchinson's Estimator, the partial trace operator, and the partial transpose operator. Using this equation, we show that when using real vectors, in the worst case, this estimator needs $O(\frac{3^k}{\varepsilon^2})$ products to recover a $(1\pm\varepsilon)$ approximation of the trace of any PSD $\mathbf{A}$, and a matching lower bound for certain PSD $\mathbf{A}$. However, when using complex vectors, this can be exponentially improved to $\Theta(\frac{2^k}{\varepsilon^2})$. Further, if the $\mathrm{x}_i$ vectors are low-dimensional and if we instead build $\mathrm{x}$ as the Kronecker product of (scaled) random unit vectors on the complex sphere, then as few as $\frac{1.33^k}{\varepsilon^2}$ samples suffice. We show that Hutchinson's Estimator converges slowest when $\mathbf{A}$ itself also has Kronecker structure. We conclude with some theoretical evidence suggesting that, by combining Hutchinson's Estimator with other techniques, it may be possible to avoid the exponential dependence on $k$.