On Sketching Trimmed Statistics

TL;DR

This paper introduces space-efficient linear sketching methods for estimating trimmed statistics of frequency vectors, enabling robust, fast, and memory-efficient analysis in streaming and distributed data settings.

Contribution

It provides new conditions and algorithms for sketching trimmed frequency statistics, including optimal error guarantees and extensions to various related problems.

Findings

Achieves poly(1/ε, log n) space for approximating top-k Fp moments when k ≥ n/polylog n.

Establishes necessary conditions relating the k-th largest frequency to tail mass for general k.

Empirically demonstrates reduced space usage compared to Count-Sketch with comparable accuracy.

Abstract

We present space-efficient linear sketches for estimating trimmed statistics of an $n$-dimensional frequency vector $x$, e.g., the sum of $p$-th powers of the largest $k$ frequencies (i.e., entries) in absolute value, or the $k$-trimmed vector, which excludes the top and bottom $k$ frequencies. This is called the $F_p$ moment of the trimmed vector. Trimmed measures are used in robust estimation, as seen in the R programming language's `trim.var' function and the `trim' parameter in the mean function. Linear sketches improve time and memory efficiency and are applicable to streaming and distributed settings. We initiate the study of sketching these statistics and give a new condition for capturing their space complexity. When $k \ge n/poly\log n$, we give a linear sketch using $poly(1/\varepsilon, \log n)$ space which provides a $(1 \pm \varepsilon)$ approximation to the top-$k$ $F_p$ moment for $p \in [0,2]$. For general $k$, we give a sketch with the same guarantees under a condition relating the $k$-th largest frequency to the tail mass, and show this condition is necessary. For the $k$-trimmed version, our sketch achieves optimal error guarantees under the same condition. We extend our methods to $p > 2$ and also address related problems such as computing the $F_p$ moment of frequencies above a threshold, finding the largest $k$ such that the $F_p$ moment of the top $k$ exceeds $k^{p+1}$, and the $F_p$ moment of the top $k$ frequencies such that each entry is at least $k$. Notably, our algorithm for this third application improves upon the space bounds of the algorithm of Govindan, Monemizadeh, and Muthukrishnan (PODS '17) for computing the $h$-index. We show empirically that our top $k$ algorithm uses much less space compared to Count-Sketch while achieving the same error.