Subspace exploration: Bounds on Projected Frequency Estimation

TL;DR

This paper investigates the space complexity of computing data analysis functions over subspaces of high-dimensional data, establishing lower bounds and proposing upper bounds that improve upon naive approaches.

Contribution

It provides new lower bounds and space-approximation tradeoffs for subspace frequency estimation problems, using coding theory and combinatorial reductions.

Findings

Many subspace analysis problems require exponential space in the dimension.

Efficient approximations are achievable with sub-exponential space complexity.

The results demonstrate significant improvements over naive methods for high-dimensional data analysis.

Abstract

Given an $n \times d$ dimensional dataset $A$, a projection query specifies a subset $C \subseteq [d]$ of columns which yields a new $n \times |C|$ array. We study the space complexity of computing data analysis functions over such subspaces, including heavy hitters and norms, when the subspaces are revealed only after observing the data. We show that this important class of problems is typically hard: for many problems, we show $2^{\Omega(d)}$ lower bounds. However, we present upper bounds which demonstrate space dependency better than $2^d$. That is, for $c,c' \in (0,1)$ and a parameter $N=2^d$ an $N^c$-approximation can be obtained in space $\min(N^{c'},n)$, showing that it is possible to improve on the na\"{i}ve approach of keeping information for all $2^d$ subsets of $d$ columns. Our results are based on careful constructions of instances using coding theory and novel combinatorial reductions that exhibit such space-approximation tradeoffs.