Tight Bounds for the Subspace Sketch Problem with Applications

TL;DR

None

Contribution

None

Abstract

In the subspace sketch problem one is given an $n\times d$ matrix $A$ with $O(\log(nd))$ bit entries, and would like to compress it in an arbitrary way to build a small space data structure $Q_p$, so that for any given $x \in \mathbb{R}^d$, with probability at least $2/3$, one has $Q_p(x)=(1\pm\epsilon)\|Ax\|_p$, where $p\geq 0$, and where the randomness is over the construction of $Q_p$. The central question is: How many bits are necessary to store $Q_p$? This problem has applications to the communication of approximating the number of non-zeros in a matrix product, the size of coresets in projective clustering, the memory of streaming algorithms for regression in the row-update model, and embedding subspaces of $L_p$ in functional analysis. A major open question is the dependence on the approximation factor $\epsilon$. We show if $p\geq 0$ is not a positive even integer and $d=\Omega(\log(1/\epsilon))$, then $\tilde{\Omega}(\epsilon^{-2}d)$ bits are necessary. On the other hand, if $p$ is a positive even integer, then there is an upper bound of $O(d^p\log(nd))$ bits independent of $\epsilon$. Our results are optimal up to logarithmic factors, and show in particular that one cannot compress $A$ to $O(d)$ "directions" $v_1,\dots,v_{O(d)}$, such that for any $x$, $\|Ax\|_1$ can be well-approximated from $\langle v_1,x\rangle,\dots,\langle v_{O(d)},x\rangle$. Our lower bound rules out arbitrary functions of these inner products (and in fact arbitrary data structures built from $A$), and thus rules out the possibility of a singular value decomposition for $\ell_1$ in a very strong sense. Indeed, as $\epsilon\to 0$, for $p = 1$ the space complexity becomes arbitrarily large, while for $p = 2$ it is at most $O(d^2 \log(nd))$. As corollaries of our main lower bound, we obtain new lower bounds for a wide range of applications, including the above, which in many cases are optimal.

Equations

$$\mathrm{cost}(X,\mathcal{C})=\sum _{x\in X}\varphi (\mathrm{dist}(x,\mathcal{C})),$$

$$\underset{y}{\min }\left(\Phi (Ay)+\lambda \Phi (x-y)\right)=\Phi (Ax).$$

$$(1-\epsilon )\Vert Ax{\Vert }_p\le \Vert TAx{\Vert }_p\le (1+\epsilon )\Vert Ax{\Vert }_p.$$

$$N_p(d,\epsilon )\le \{\begin{array}{ll}C{\epsilon }^{-2}d\log d& p=1\\ C{\epsilon }^{-2}d(\log {\epsilon }^{-2}d)(\log \log {\epsilon }^{-2}d+\log (1/\epsilon ){)}^2& p\in (1,2)\\ C_p{\epsilon }^{-2}{d}^{p/2}{\log }^{2}d\log (d/\epsilon )& p\in (2,\infty )\setminus 2\mathbb{Z}\\ C{\epsilon }^{-2}(10d/p{)}^{p/2}& p\in 2\mathbb{Z},\end{array}$$

$$(1-\epsilon )C(S,V\setminus S)\le f(S)\le (1+\epsilon )C(S,V\setminus S),$$

$$\Vert Ai{\Vert }_1=\sum _{j\in \{-1,1{\}}^d}{r}_j\cdot |⟨i,j⟩|\le 2^d\cdot \mathrm{poly}(d)<\frac{1}{\epsilon }.$$

$$\left|\sum _{\begin{array}{c}0\le i\le d\\ \text{i is even}\end{array}}(-1{)}^{i/2}(\genfrac{}{}{0ex}{}{d/2}{i/2})|d-2i|\right|,$$

$$r_i=\{\begin{array}{ll}{s}_i,& i\le \mathrm{rank}(M);\\ 0,& i>\mathrm{rank}(M),\end{array}$$

$$\left|\sum _{\begin{array}{c}0\le i\le d\\ \text{i is even}\end{array}}(-1{)}^{i/2}(\genfrac{}{}{0ex}{}{d/2}{i/2})|d-2i{|}^p\right|=\Theta \left(\frac{2^{d/2}}{\sqrt{d}}\left|\sin \frac{p\pi }{2}\right|\right).$$

$$r=\sum _{i=1}^n{s}_i\cdot \frac{M_i}{\Vert M_i{\Vert }_2},$$

$$⟨M_i,r⟩=s_i\cdot \Vert M_i{\Vert }_2.$$

$$\Vert At{\Vert }_p^p\le n\cdot (d\log d{)}^{p/2}\ll d^p$$

$$\kappa (A)=\frac{{\sup }_{\Vert x{\Vert }_2=1}\Vert Ax{\Vert }_2}{{\inf }_{\Vert x{\Vert }_2=1}\Vert Ax{\Vert }_2}.$$

$$\Vert A-B{\Vert }_F^2\ge \sum _{i=k+1}^r{\sigma }_i^2.$$

$$\mathrm{Pr}\left\{\sum _i{s}_i{a}_i>t\right\}\le \exp \left(-\frac{t^2}{2{\sum }_i{a}_i^2}\right).$$

$$A{\left(\sum _i{a}_i^2\right)}^{1/2}\le {\left(E|\sum _i{s}_i{a}_i{|}^p\right)}^{1/p}\le B\sqrt{p}{\left(\sum _i{a}_i^2\right)}^{1/2}.$$

$$\mathrm{Pr}\left\{f(X)-Ef(X)\ge t\right\}\le e^{-t^2/8}.$$

$$\mathrm{Pr}\left\{|f(x)-(E(f(x){)}^p{)}^{1/p}|\ge t\right\}\le 2e^{-c{t}^2},$$

$$\sqrt{N}-C\sqrt{n}-t\le {\sigma }_{\min }(A)\le {\sigma }_{\max }(A)\le \sqrt{N}+C\sqrt{n}+t,$$

$$\mathrm{Pr}\left\{{\Vert \sum _{i=1}^k{s}_i\cdot U_i\Vert }_{\infty }\le 3\sqrt{\ln k}\right\}\ge 1-\frac{1}{k^{1.3}}.$$

$$Z_j=\sum _{i=1}^k{s}_i{U}_{i,j}.$$

$$\sum _{i=1}^k{U}_{i,j}^2\le 1.$$

$$\mathrm{Pr}\{|Z_j|\ge 3\sqrt{\ln k}\}\le \exp (-2\ln k).$$

$$E_n(f;[a,b])=\underset{p\in P_n}{\inf }\Vert f-p{\Vert }_{\infty },$$

$$E_n(f;[-1,1])\le \frac{6^{k+1}{e}^k}{(k+1)n^k}{\omega }_k\left(\frac{1}{n-k}\right),$$

$${\omega }_k(\delta )=\underset{\begin{array}{c}x,y\in [-1,1]\\ |x-y|\le \delta \end{array}}{\sup }\left|f^{(k)}(x)-f^{(k)}(y)\right|.$$

$$M_{i,j}^{(d,p)}=|⟨i,j⟩{|}^p,$$

$$\left|\hat{g}(s)\right|=\left|\sum _{x\in {\mathbb{F}}_2^d}(-1{)}^{⟨s,x⟩}g(x)\right|,s\in {\mathbb{F}}_2^d.$$

$$(H_s^{(d)}{)}_z=\frac{1}{2^{d/2}}(-1{)}^{⟨s,z⟩},s,z\in {\mathbb{F}}_2^d,$$

$$\left|\sum _{\begin{array}{c}0\le i\le d\\ \text{i is even}\end{array}}(-1{)}^{i/2}(\genfrac{}{}{0ex}{}{d/2}{i/2})|d-2i{|}^p\right|\triangleq {\Lambda }_0^{(d,p)}\ge 0.$$

Full text

How many bits are necessary to store $Q_{p}$?

This problem has applications to the communication of approximating the number of non-zeros in a matrix product, the size of coresets in projective clustering, the memory of streaming algorithms for regression in the row-update model, and embedding subspaces of $L_{p}$ in functional analysis. A major open question is the dependence on the approximation factor $\varepsilon$.

We show if $p\geq 0$ is not a positive even integer and $d=\Omega(\log(1/\varepsilon))$, then $\widetilde{\Omega}(\varepsilon^{-2}\cdot d)$ bits are necessary. On the other hand, if $p$ is a positive even integer, then there is an upper bound of $O(d^{p}\log(nd))$ bits independent of $\varepsilon$. Our results are optimal up to logarithmic factors, and show in particular that one cannot compress $A$ to $O(d)$ “directions” $v_{1},\ldots,v_{O(d)}$, such that for any $x$, $\|Ax\|_{1}$ can be well-approximated from $\langle v_{1},x\rangle,\ldots,\langle v_{O(d)},x\rangle$. Our lower bound rules out arbitrary functions of these inner products (and in fact arbitrary data structures built from $A$), and thus rules out the possibility of a singular value decomposition for $\ell_{1}$ in a very strong sense. Indeed, as $\varepsilon\rightarrow 0$, for $p=1$ the space complexity becomes arbitrarily large, while for $p=2$ it is at most $O(d^{2}\log(nd))$. As corollaries of our main lower bound, we obtain new lower bounds for a wide range of applications, including the above, which in many cases are optimal.

1. 1 Introduction

2. 1.1 Our Contributions

3. 1.2 Connection with Banach Space Theory

4. 1.3 Comparison with Prior Work

5. 1.3.1 Comparison with Previous Results in Functional Analysis

6. 1.3.2 Comparison with Previous Results for Graph Sparsifiers

7. 1.4 Our Techniques

8. 2 Preliminaries

9. 3 An $\widetilde{\Omega}\left(\varepsilon^{-2}\right)$ Lower Bound

10. 3.1 Spectrum of Matrices $M$

11. 3.2 Orthogonalizing Rows

12. 3.3 Space Lower Bound on $Q_{p}$

13. 4 Lower Bounds for $p>2$

14. 4.1 Lower Bounds for the Subspace Sketch Problem for $p>2$

15. 4.2 Lower Bounds for the For-All Version

16. 4.2.1 Lower Bound for $p\geq 2$

17. 4.2.2 Lower Bound for $1\leq p\leq 2$

18. 5 Linear Embeddings

19. 6 Sampling-based Embeddings

20. 7 Oblivious Sketches

21. 8 Lower Bounds for $M$-estimators

22. 8.1 Proof of Lemma 8.3

23. 9 Lower Bounds on Coresets for Projective Clustering

24. 10 Upper Bounds for the Tukey Loss $p$-Norm

25. 10.1 Mollification of Tukey Loss Function

26. 10.2 Estimation Algorithm

27. 11 An Upper Bound for $\ell_{1}$ Subspace Sketches in Two Dimensions

1 Introduction

The explosive growth of available data has necessitated new models for processing such data. A particularly powerful tool for analyzing such data is sketching, which has found applications to communication complexity, data stream algorithms, functional analysis, machine learning, numerical linear algebra, sparse recovery, and many other areas. Here one is given a large object, such as a graph, a matrix, or a vector, and one seeks to compress it while still preserving useful information about the object. One of the main goals of a sketch is to use as little memory as possible in order to compute functions of interest. Typically, to obtain non-trivial space bounds, such sketches need to be both randomized and approximate. By now there are nearly-optimal bounds on the memory required of sketching many fundamental problems, such as graph sparsification, norms of vectors, and problems in linear algebra such as low-rank approximation and regression. We refer the reader to the surveys [30, 40] as well as the compilation of lecture notes [2].

In this paper we consider the subspace sketch problem.

Definition 1.1**.**

Given an $n\times d$ matrix $A$ with entries specified by $O(\log(nd))$ bits, an accuracy parameter $\varepsilon>0$, and a function $\Phi:\mathbb{R}^{n}\rightarrow\mathbb{R}^{\geq 0}$, design a data structure $Q_{\Phi}$ so that for any $x\in\mathbb{R}^{d}$, with probability at least $0.9$, $Q_{\Phi}(x)=(1\pm\varepsilon)\Phi(Ax)$.

The subspace sketch problem captures many important problems as special cases. We will show how to use this problem to bound the communication of approximating statistics of a matrix product, the size of coresets in projective clustering, the memory of streaming algorithms for regression in the row-update model, and the embedding dimension in functional analysis. We will describe these applications in more detail below.

The goal in this work is to determine the memory, i.e., the size of $Q_{\Phi}$, required for solving the subspace sketch problem for different functions $\Phi$. We first consider the classical $\ell_{p}$-norms $\Phi(x)=\sum_{i=1}^{n}|x_{i}|^{p}$, in which case the problem is referred to as the $\ell_{p}$ subspace sketch problem111Note we are technically considering the $p$-th power of the $\ell_{p}$-norms, but for the purposes of $(1+\varepsilon)$-approximation, they are the same for constant $p$. Also, when $p<1$, $\ell_{p}$ is not a norm, though it is still a well-defined quantity. Finally, $\ell_{0}$ denotes the number of non-zero entries of $x$.. We later extend our techniques to their robust counterparts $\Phi(x)=\sum_{i=1}^{n}\phi(x_{i})$, where $\phi(t)=|t|^{p}$ if $|t|\leq\tau$ and $\phi(t)=\tau^{p}$ otherwise. Here $\Phi$ is a so-called $M$-estimator and known as the Tukey loss $p$-norm. It is less sensitive to “outliers” since it truncates large coordinate valus at $\tau$. We let $Q_{p}$ denote $Q_{\Phi}$ when $\Phi(x)=\sum_{i}|x_{i}|^{p}$, and use $Q_{p,\tau}$ when $\Phi$ is the Tukey loss $p$-norm.

It is known that for $p\in(0,2]$ and $r=O(\varepsilon^{-2})$, if one chooses a matrix $S\in\mathbb{R}^{r\times n}$ of i.i.d. $p$-stable random variables, then for any fixed $y\in\mathbb{R}^{n}$, from the sketch $S\cdot y$ one can output a number $z$ for which $(1-\varepsilon)\|y\|_{p}\leq z\leq(1+\varepsilon)\|y\|_{p}$ with probability at least $0.9$ [17]. We say $z$ is a $(1\pm\varepsilon)$-approximation of $\|y\|_{p}$. For $p=1$, the output is just $\operatorname{med}(Sy)$, where $\operatorname{med}(\cdot)$ denotes the median of the absolute values of the coordinates in a vector. A sketch $S$ with $r=O(\varepsilon^{-2}\log n)$ rows is also known for $p=0$ [22]. For $p>2$, there is a distribution on $S\in\mathbb{R}^{r\times n}$ with $r=O(n^{1-2/p}\log n/\varepsilon^{2})$ for which one can output a $(1\pm\varepsilon)$-approximation of $\|y\|_{p}$ given $Sy$ with probability at least $0.9$ [15]. By appropriately discretizing the entries, one can solve the $\ell_{p}$ subspace sketch problem by storing $S\cdot A$ for an appropriate sketching matrix $S$, and estimating $\|Ax\|_{p}$ using $S\cdot A\cdot x$. In this way, one obtains a sketch of size $\widetilde{O}(\varepsilon^{-2}d)$222Throughout we use $\widetilde{O},\widetilde{\Omega},$ and $\widetilde{\Theta}$ to hide factors that are polynomial in $\log(nd/\varepsilon)$. We note that our lower bounds are actually independent of $n$. bits for $p\in[0,2]$, and a sketch of size $\widetilde{O}(n^{1-2/p}/\varepsilon^{2}\cdot d)$ bits for $p>2$. Note, however, that this is only one particular approach, based on choosing a random matrix $S$, and better approaches may be possible. Indeed, note that for $p=2$, one can simply store $A^{T}A$ and output $Q_{2}(x)=x^{T}A^{T}Ax$. This is exact (i.e., holds for $\varepsilon=0$) and only uses $O(d^{2}\log(nd))$ bits of space, which is significantly smaller than $\widetilde{O}(\varepsilon^{-2}d)$ for small enough $\varepsilon$. We note that the $\varepsilon^{-2}$ term may be extremely prohibitive in applications. For example, if one wants high accuracy such as $\varepsilon=0.1\%$, the $\varepsilon^{-2}$ factor is a severe drawback of existing algorithms.

A natural question is what makes it possible for $p=2$ to obtain $\widetilde{O}(d^{2})$ bits of space, and whether it is also possible to achieve $\widetilde{O}(d^{2})$ space for $p=1$. One thing that makes this possible for $p=2$ is the singular value decomposition (SVD), namely, that $A=U\Sigma V^{T}$ for matrices $U\in\mathbb{R}^{n\times d}$ and $V\in\mathbb{R}^{d\times d}$ with orthonormal columns, and $\Sigma$ is a non-negative diagonal matrix. Then $\|Ax\|_{2}^{2}=\|\Sigma V^{T}x\|_{2}^{2}$ since $U$ has orthonormal columns. Consequently, it suffices to maintain the $d$ inner products $\langle\Sigma_{1,1}v_{1},x\rangle,\ldots,\langle\Sigma_{d,d}v_{d},x\rangle$, where the $v_{i}$’s are the rows of $V^{T}$. Thus one can “compress” $A$ to $d$ “directions” $\Sigma_{i,i}v_{i}$. A natural question is whether for $p=1$ it is also possible find $O(d)$ directions $v_{1},\ldots,v_{O(d)}$, such that for any $x$, $\|Ax\|_{1}$ can be well-approximated from some function of $\langle v_{1},x\rangle,\ldots,\langle v_{O(d)},x\rangle$. Indeed, this would be the analogue of the SVD for $p=1$, for which little is known.

The central question of our work is:

How much memory is needed to solve the subspace sketch problem as a function of $\Phi$?

1.1 Our Contributions

Up to polylogarithmic factors, we resolve the above question for $\ell_{p}$-norms and Tukey loss $p$-norms for any $p\in[0,2)$. For $p\geq 2$ we also obtain a surprising separation for even integers $p$ from other values of $p$.

Our main theorem is the following. We denote by $\mathbb{Z}^{+}$ the set of positive integers.

Theorem 1.1** (Informal).**

Let $p\in[0,\infty)\setminus 2\mathbb{Z}^{+}$ be a constant. For any $d=\Omega(\log(1/\varepsilon))$ and $n=\widetilde{\Omega}(\varepsilon^{-2}\cdot d)$, we have that $\widetilde{\Omega}(\varepsilon^{-2}\cdot d)$ bits are necessary to solve the $\ell_{p}$ subspace sketch problem.

When $p\in 2\mathbb{Z}^{+}$, there is an upper bound of $O(d^{p}\log(nd))$ bits, independent of $\varepsilon$ (see Remark 3.15). This gives a surprising separation between positive even integers and other values of $p$; in particular for positive even integers $p$ it is possible to obtain $\varepsilon=0$ with at most $O(d^{p}\log(nd))$ bits of space, whereas for other values of $p$ the space becomes arbitrarily large as $\varepsilon\rightarrow 0$. This also shows it is not possible, for $p=1$ for example, to find $O(d)$ representative directions for $\varepsilon=0$ analogous to the SVD for $p=2$. Note that the lower bound in Theorem 1.1 is much stronger than this, showing that there is no data structure whatsoever which uses fewer than $\widetilde{\Omega}(\varepsilon^{-2}\cdot d)$ bits, and so as $\varepsilon$ gets smaller, the space complexity becomes arbitrarily large.

In addition to the $\ell_{p}$-norm, in the subspace sketch problem we also consider a more general entry-decomposable $\Phi$, that is, $\Phi(v)=\sum_{i}\phi(v_{i})$ for $v\in\mathbb{R}^{n}$ and some $\phi:\mathbb{R}\to\mathbb{R}^{\geq 0}$. We show the same $\widetilde{\Omega}(\varepsilon^{-2}\cdot d)$ lower bounds for a number of $M$-estimators $\phi$.

Theorem 1.2**.**

The subspace sketch problem requires $\widetilde{\Omega}(\varepsilon^{-2}d)$ bits when $d=\Omega(\log(1/\varepsilon))$ and $n=\widetilde{\Omega}(\varepsilon^{-2}\cdot d)$ for the following functions $\phi$:

• •

($L_{1}$-$L_{2}$* estimator) $\phi(t)=2(\sqrt{1+t^{2}/2}-1)$;*

• •

(Huber estimator) $\phi(t)=t^{2}/(2\tau)\cdot\mathbf{1}_{\{|t|\leq\tau\}}+(|t|-\tau/2)\cdot\mathbf{1}_{\{|t|>\tau\}}$;

• •

(Fair estimator) $\phi(t)=\tau^{2}(|x|/\tau-\ln(1+|t|/\tau))$;

• •

(Cauchy estimator) $\phi(t)=(\tau^{2}/2)\ln(1+(t/\tau)^{2})$;

• •

(Tukey loss $p$-norm) $\phi(t)=|t|^{p}\cdot\mathbf{1}_{\{|t|\leq\tau\}}+\tau^{p}\cdot\mathbf{1}_{\{|t|>\tau\}}$.

We also consider a mollified version of the $1$-Tukey loss function, for which the lower bound of $\widetilde{\Omega}(\varepsilon^{-2}d)$ bits still holds. Furthermore, the lower bound is tight up to logarithmic factors, since we design a new algorithm which approximates $\Phi(x)$ using $\widetilde{O}(\varepsilon^{-2})$ bits, which implies an upper bound of $\widetilde{O}(\varepsilon^{-2}d)$ for the subspace sketch problem. See Section 10 for details.

While Theorem 1.1 gives a tight lower bound for $p\in[0,2)$, matching the simple sketching upper bounds described earlier, and also gives a separation from the $O(d^{p}\log(nd))$ bit bound for even integers $p\geq 2$, one may ask what exactly the space required is for even integers $p\geq 2$ and arbitrarily small $\varepsilon$. For $p=2$, the $O(d^{2}\log(nd))$ upper bound is tight up to logarithmic factors since the previous work [3, Theorem 2.2] implies an $\widetilde{\Omega}(d^{2})$ lower bound once $\varepsilon=O(1/\sqrt{d})$. For $p>2$, we show the following: for a constant $\varepsilon\in(0,1)$, there is an upper bound of $\widetilde{O}(d^{p/2})$ bits (see Remark 4.4), which is nearly tight in light of the following lower bound, which holds for constant $\varepsilon$.

Theorem 1.3** (Informal).**

Let $p\geq 2$ and $\varepsilon\in(0,1)$ be constants. Suppose that $n=\widetilde{\Omega}(d^{p/2})$, then $\widetilde{\Omega}(d^{p/2})$ bits are necessary to solve the $\ell_{p}$ subspace sketch problem.

Note that Theorem 1.3 holds even if $p$ is not an even integer, and shows that a lower bound of $d^{\Omega(p)}$ holds for every $p\geq 2$.

We next turn to concrete applications of Theorems 1.1 and 1.2.

Statistics of a Matrix Product.

In [41], an algorithm was given for estimating $\|A\cdot B\|_{p}$ for integer matrices $A$ and $B$ with $O(\log n)$ bit integer entries (see Algorithm 1 in [41] for the general algorithm). When $p=0$, this estimates the number of non-zero entries of $A\cdot B$, which may be useful since there are faster algorithms for matrix product when the output is sparse, see [31] and the references therein. More generally, norms of the product $A\cdot B$ can be used to determine how correlated the rows of $A$ are with the columns of $B$. The bit complexity of this problem was studied in [38, 41]. In [38] a lower bound of $\Omega(\varepsilon^{-2}\cdot n)$ bits was shown for estimating $\|AB\|_{0}$ for $n\times n$ matrices $A,B$ up to a $(1+\varepsilon)$ factor, assuming $n\geq 1/\varepsilon^{2}$ (this lower bound holds already for binary matrices $A$ and $B$). This lower bound implies an $\ell_{0}$-subspace sketch lower bound of $\Omega(\varepsilon^{-2}\cdot d)$ assuming that $d\geq 1/\varepsilon^{2}$. Our lower bound in Theorem 1.1 considerably strengthens this result by showing the same lower bound (up to $\operatorname{polylog}(d/\varepsilon)$ factors) for a much smaller value of $d=\Omega(\log(1/\varepsilon))$. For any $p\in[0,2]$, there is a matching upper bound up to polylogarithmic factors (such an upper bound is given implicitly in the description of Algorithm 1 of [41], where the $\varepsilon$ there is instantiated with $\sqrt{\varepsilon}$, and also follows from the random sketching matrices $S$ discussed above).

Projective Clustering.

In the task of projective clustering, we are given a set $X\subset\mathbb{R}^{d}$ of $n$ points, a positive integer $k$, and a non-negative integer $j\leq d$. A center $\mathcal{C}$ is a $k$-tuple $(V_{1},V_{2},\ldots,V_{k})$, where each $V_{i}$ is a $j$-dimensional affine subspace in $\mathbb{R}^{d}$. Given a function $\phi:\mathbb{R}\to\mathbb{R}^{\geq 0}$, the objective is to find a center $\mathcal{C}$ that minimizes the projective cost, defined to be

[TABLE]

where $\operatorname{dist}(x,\mathcal{C})=\min_{i}\operatorname{dist}(x,V_{i})$, the Euclidean distance from a point $p$ to its nearest subspace $V_{i}$ in $\mathcal{C}=(V_{1},V_{2},\ldots,V_{k})$. The coreset problem for projective clustering asks to design a data structure $Q_{\phi}$ such that for any center $\mathcal{C}$, with probability at least $0.9$, $Q_{\phi}(\mathcal{C})=(1\pm\varepsilon)\operatorname{cost}(X,\mathcal{C})$. Note that in this and other computational geometry problems, the dimension $d$ may be small (e.g., $d=\log(1/\varepsilon$)), though one may want a high accuracy solution. Although possibly far from optimal, surprisingly our lower bound below is the first non-trivial lower bound on the size of coresets for projective clustering.

Theorem 1.4** (Informal).**

Suppose that $\phi(t)=|t|^{p}$ for $p\in[0,\infty)\setminus 2\mathbb{Z}^{+}$ or $\phi$ is one of the functions in Theorem 1.2. For $k\geq 1$ and $j=\Omega(\log(k/\varepsilon))$, any coreset for projective clustering requires $\widetilde{\Omega}(\varepsilon^{-2}kj)$ bits.

Linear Regression.

In the linear regression problem, there is an $n\times d$ data matrix $A$ and a vector $b\in\mathbb{R}^{n}$. The goal is to find a vector $x\in\mathbb{R}^{d}$ so as to minimize $\Phi(Ax-b)$, where $\Phi(v)=\sum_{i}\phi(v_{i})$ for $v\in\mathbb{R}^{n}$ and some $\phi:\mathbb{R}\to\mathbb{R}^{\geq 0}$. Here we consider streaming coresets for linear regression in the row-update model. In the row-update model, the streaming coreset is updated online during one pass over the $n$ rows of $\begin{pmatrix}A&b\end{pmatrix}$, and outputs a $(1\pm\varepsilon)$-approximation to the optimal value $\min_{x}\Phi(Ax-b)$ at the end. By a simple reduction, our lower bound for the subspace sketch problem implies lower bounds on the size of streaming coresets for linear regression in the row-update model. To see this, we note that by taking sufficiently large $\lambda$,

[TABLE]

Thus, a streaming coreset for linear regression can solve the subspace sketch problem, which we formalize in the following corollary.

Corollary 1.5**.**

Suppose that $\phi(t)=|t|^{p}$ for $p\in[0,\infty)\setminus 2\mathbb{Z}^{+}$ or $\phi$ is one of the functions in Theorem 1.2. Any streaming coreset for linear regression in the row-update model requires $\widetilde{\Omega}\left(\varepsilon^{-2}d\right)$ bits when $d=\Omega(\log(1/\varepsilon))$.

Subspace Embeddings.

Let $p\geq 1$. Given $A\in\mathbb{R}^{n\times d}$, the $\ell_{p}$ subspace embedding problem asks to find a linear map $T:\mathbb{R}^{n}\to\mathbb{R}^{r}$ such that for all $x\in\mathbb{R}^{d}$,

[TABLE]

The smallest $r$ which admits a $T$ for every $A$ is denoted by $N_{p}(d,\varepsilon)$, which is of great interest in functional analysis. When $T$ is allowed to be random, we require (1) to hold with probability at least $0.9$. This problem can be seen as a special case of the “for-all” version of the subspace sketch problem in Definition 1.1. In the for-all version of the subspace sketch problem, the data structure $Q_{p}$ is required to, with probability at least $0.9$, satisfy $Q_{p}(x)=(1\pm\varepsilon)\|Ax\|_{p}$ simultaneously for all $x\in\mathbb{R}^{d}$. In this case, the same lower bound of $\widetilde{\Omega}(\varepsilon^{-2}\cdot d)$ bits holds for $p\in[1,\infty)\setminus 2\mathbb{Z}$.

Since the data structure can store $T$ if it exists, we can turn our bit lower bound into a dimension lower bound on $N_{p}(d,\varepsilon)$. Doing so will incur a loss of an $\widetilde{O}(d)$ factor (Theorem 5.1). We give an $\widetilde{\Omega}(\varepsilon^{-2})$ lower bound, which is the first such lower bound giving a dependence on $\varepsilon$ for general $p$.

Corollary 1.6**.**

Suppose that $p\in[1,\infty)\setminus 2\mathbb{Z}$ and $d=\Omega(\log(1/\varepsilon))$. It holds that $N_{p}(d,\varepsilon)=\widetilde{\Omega}(\varepsilon^{-2})$.

The dependence on $\varepsilon$ in this lower bound is tight, up to $\operatorname{polylog}(1/\varepsilon)$ factors, for all values of $p\in[1,\infty)\setminus 2\mathbb{Z}$  [34]. When $p\in 2\mathbb{Z}$, no lower bound with a dependence on $\varepsilon$ should exist, since a $d$-dimensional subspace of $\ell_{p}^{n}$ always embeds into $\ell_{p}^{r}$ isometrically with $r=\binom{d+p-1}{p}-1$ [23]. See more discussion below in Section 1.2 on functional analysis. We also prove a bit complexity lower bound for the aforementioned for-all version of the subspace sketch problem. We refer the reader to Section 4.2 for details.

Theorem 1.7**.**

Let $p\geq 1$ be a constant. Suppose that $\varepsilon>0$ is a constant. The for-all version of the subspace sketch problem requires $\Omega(d^{\max\{p/2,1\}+1})$ bits.

This lower bound immediately implies a dimension lower bound of $N_{p}(d,\varepsilon)=\widetilde{\Omega}(d^{\max\{p/2,1\}})$ for the subspace embedding problem for constant $\varepsilon$, recovering existing lower bounds (up to logarithmic factors), which are known to be tight.

Sampling by Lewis Weights.

While it is immediate that $N_{p}(d,\varepsilon)\geq d$, our lower bound above thus far has not precluded the possibility that $N_{p}(d,\varepsilon)=\widetilde{O}(d+1/\varepsilon^{2})$. However, the next corollary, which lower bounds the target dimension for sampling-based embeddings, indicates this is impossible to achieve using a prevailing existing technique.

Corollary 1.8**.**

Let $p\geq 1$ and $p\notin 2\mathbb{Z}$. Suppose that $Q_{p}(x)=\|TAx\|_{p}^{p}$ solves the $\ell_{p}$ subspace sketch problem for some $T\in\mathbb{R}^{r\times n}$ for which each row of $T$ contains exactly one non-zero element. Then $r=\widetilde{\Omega}(\varepsilon^{-2}d)$, provided that $d=\Omega(\log(1/\varepsilon))$.

The same lower bound holds for the for-all version of the $\ell_{p}$ subspace sketch problem. As a consequence, since the upper bounds of $N_{p}(d,\varepsilon)$ in (2) for $1\leq p<2$ are based on subsampling with the “change of density” technique (also known as sampling by Lewis weights [13]), they are, within the framework of this classical technique, best possible up to polylog$(d/\varepsilon)$ factors.

Oblivious Sketches.

For the for-all version of the $\ell_{p}$ subspace sketch problem, we note that there exist general sketches such as the Cauchy sketch [12] which are beyond the reach of the corollary above. Note that the Cauchy sketch is an oblivious sketch, which means the distribution is independent of $A$. We also prove a dimension lower bound of $\widetilde{\Omega}(\varepsilon^{-2}\cdot d)$ on the target dimension for oblivious sketches (see Section 7), which is tight up to logarithmic factors since the Cauchy sketch has a target dimension of $O(\varepsilon^{-2}d\log(d/\varepsilon))$.

Theorem 1.9** (Informal).**

Let $p\in[1,2)$ be a constant. Any oblivious sketch that solves the for-all version of the $\ell_{p}$ subspace sketch problem has a target dimension of $\widetilde{\Omega}(\varepsilon^{-2}d)$.

Therefore, it is natural to ask in general whether $N_{p}(d,\varepsilon)=\widetilde{\Omega}(d/\varepsilon^{2})$. A proof using the framework of this paper would require an $\widetilde{\Omega}(d^{2}/\varepsilon^{2})$ lower bound for the for-all version of the $\ell_{p}$ subspace sketch problem. We conjecture it is true; however, our current methods, giving almost-tight lower bounds (in the for-each sense), do not extend to give this result and so we leave it as a main open problem.

1.2 Connection with Banach Space Theory

In the language of functional analysis, the $\ell_{p}$ subspace embedding problem is a classical problem in the theory of $L_{p}$ spaces with a rich history. For two Banach spaces $X$ and $Y$, we say $X$ $K$-embeds into $Y$, if there exists an injective homomorphism $T:X\to Y$ satisfying $\|x\|_{X}\leq\|Tx\|_{Y}\leq K\|x\|_{X}$ for all $x\in X$. Such a $T$ is called an isomorphic embedding. A classical problem in the theory of Banach spaces is to consider the isomorphic embedding of finite-dimensional subspaces of $L_{p}=L^{p}(0,1)$ into $\ell_{p}^{n}=(\mathbb{R}^{n},\|\cdot\|_{p})$, where $p\geq 1$ is a constant. Specifically, the problem asks what is the minimum value of $n$, denoted by $N_{p}(d,\varepsilon)$, for which all $d$-dimensional subspaces of $L_{p}$ $(1+\varepsilon)$-embed into $\ell_{p}^{n}$. A comprehensive survey of this problem can be found in [20].

The case of $p=2$ is immediate, in which case one can take $n=d$ and $\varepsilon=0$, obtaining an isometric embedding, and thus we assume $p\neq 2$. We remark that, when $p$ is an even integer, it is also possible to attain an isometric embedding into $\ell_{p}^{n}$ with $n=\binom{d+p-1}{p}-1$ [23]. In general, the best333A few upper bounds for the case $p\in(2,\infty)\setminus 2\mathbb{Z}$ are known, none of which dominates the rest. Here we choose the one having the best dependence on both $d$ and $\varepsilon$, up to $\operatorname{polylog}(d/\varepsilon)$ factors. known upper bounds on $N_{p}(d,\varepsilon)$ are as follows.

[TABLE]

where $C>0$ is an absolute constant and $C_{p}>0$ is a constant that depends only on $p$. The cases of $p=1$ and $p\in(1,2)$ are due to Talagrand [36, 37]. The case of non-even integers $p>2$ is taken from [24, Theorem 15.13], based on the earlier work of Bourgain et al. [7]. The case of even integers $p$ is due to Schechtman [35].

The upper bounds in (2) are established by subsampling with a technique called “change of density” [20]. First observe that it suffices to consider embeddings from $\ell_{p}^{N}$ to $\ell_{p}^{n}$ since any $d$-dimensional subspace of $L_{p}$ $(1+\varepsilon)$-embeds into $\ell_{p}^{N}$ for some large $N$. Now suppose that $E$ is a $d$-dimensional subspace of $\ell_{p}^{N}$. One can show that randomly subsampling coordinates induces a low-distortion isomorphism between $E$ and $E$ restricted onto the sampled coordinates, provided that each element of $E$ is “spread out” among the coordinates, which is achieved by first applying the technique of change of density to $E$.

Regarding lower bounds, a quick lower bound follows from the tightness of Dvoretzky’s Theorem for $\ell_{p}$ spaces (see, e.g. [28, p21]), which states that if $\ell_{2}^{d}$ $2$-embeds into $\ell_{p}^{n}$, then $n\geq cd$ for $1\leq p<2$ and $n\geq(cd/p)^{p/2}$ for $p\geq 2$, where $c>0$ is an absolute constant. Since $\ell_{2}^{d}$ embeds into $L_{p}$ isometrically for all $p\geq 1$ [19, p16], identical lower bounds for $N_{p}(d,\varepsilon)$ follow. Hence the upper bounds in (2) are, in terms of $d$, tight for $p\in 2\mathbb{Z}$ and tight up to logarithmic factors for other values of $p$. However, the right dependence on $\varepsilon$ is a long-standing open problem and little is known. See [20, p845] for a discussion on this topic. It is known that $N_{1}(d,\varepsilon)\geq c(d)\varepsilon^{-2(d-1)/(d+2)}$ [7], whose proof critically relies upon the fact that the unit ball of a finite-dimensional space of $\ell_{1}$ is the polar of a zonotope (a linear image of cube $[-1,1]^{d}$) and the $\ell_{1}$-norm for vectors in the subspace thus admits a nice representation [5]. However, a lower bound for general $p$ is unknown. Our Corollary 1.6 shows that $n\geq c\varepsilon^{-2}/\operatorname{poly}(\log(1/\varepsilon))$ for all $p\geq 1$ and $p\not\in 2\mathbb{Z}$, which is the first lower bound on the dependence of $\varepsilon$ for general $p$, and is optimal up to logarithmic factors. We would like to stress that except for the very special case of $\ell_{1}$, no lower bound on the dependence on $\varepsilon$ whatsoever was known for $p\not\in 2\mathbb{Z}$. We consider this to be significant evidence of the generality and novelty of our techniques. Moreover, even our lower bound for $p=1$ is considerably wider in scope, as discussed more below.

1.3 Comparison with Prior Work

1.3.1 Comparison with Previous Results in Functional Analysis

As discussed, the mentioned lower bounds on $N_{p}(d,\varepsilon)$ come from the tightness of Dvoretzky’s Theorem, which shows the impossibility of embedding $\ell_{2}^{d}$ into a Banach space with low distortion. Here the hardness comes from the geometry of the target space. In contrast, we emphasize that the hardness in our $\ell_{p}$ subspace sketch problem comes from the source space, since the target space is unconstrained and the output function $Q_{p}(\cdot)$ does not necessarily correspond to an embedding. The lower bound via tightness of Dvoretzky’s Theorem cannot show that $\ell_{p}^{d}$ does not $(1+\varepsilon)$-embed into $\ell_{q}^{n}$ for $d=\Theta(\log(1/\varepsilon))$ and $n=O(1/\varepsilon^{1.99})$, where $q\not\in 2\mathbb{Z}$.

When the target space is not $\ell_{p}$, lower bounds via functional analysis are more difficult to obtain since they require understanding the geometry of the target space. Since our data structure problem has no constraints on $Q_{p}(\cdot)$, the target space does not even need to be normed. In theoretical computer science and machine learning applications, the usual “sketch and solve” paradigm typically just requires the target space to admit an efficient algorithm for the optimization problem at hand444For example, consider the space $\mathbb{R}^{n}$ endowed with a premetric $d(x,y)=\sum_{i}f(x_{i}-y_{i})$, where $f(x)=\tau x\mathbf{1}_{\{x\geq 0\}}+(\tau-1)x\mathbf{1}_{\{x\leq 0\}}$ ($\tau\in(0,1)$), which is not even symmetric when $\tau\neq\frac{1}{2}$. See [42] for an embedding into this space. . Our lower bounds are thus much wider in scope than those in geometric functional analysis.

1.3.2 Comparison with Previous Results for Graph Sparsifiers

Recently, the bit complexity of cut sparsifiers was studied in [3, 10]. Given an undirected graph $G=(V,E)$, $|V|=d$, a function $f:2^{V}\to\mathbb{R}$ is a $(1+\varepsilon)$-cut sketch, if for any vertex set $S\subseteq V$,

[TABLE]

where $C(S,V\setminus S)$ denotes the capacity of the cut between $S$ and $V\setminus S$. The main result of these works is that any $(1+\varepsilon)$-cut sketch requires $\Omega(\varepsilon^{-2}d\log d)$ bits to store. Note that a cut sketch can be constructed using a for-all version of the $\ell_{p}$ subspace sketch for any $p$, by just taking the matrix $A$ to be the edge-vertex incidence matrix of the graph $G$ and querying all vectors $x\in\{0,1\}^{d}$. Thus, one may naturally ask if the lower bounds in [3, 10] imply any lower bounds for the subspace sketch problem.

We note that both works [3, 10] have explicit constraints on the value of $\varepsilon$. In [3], in order to prove the $\Omega(\varepsilon^{-2}d)$ lower bound, it is required that $\varepsilon=\Omega(1/\sqrt{d})$. In [10] the lower bound of $\Omega(d\log d/\varepsilon^{2})$ requires $\varepsilon=\omega(1/d^{1/4})$. Thus, the strongest lower bound that can be proved using such an approach is $\widetilde{\Omega}(d^{2})$. This is natural, since one can always store the entire adjacency matrix of the graph in $\widetilde{O}(d^{2})$ bits. Our lower bound, in contrast, becomes arbitrarily large as $\varepsilon\to 0$.

1.4 Our Techniques

We use the case of $p=1$ to illustrate our ideas behind the $\widetilde{\Omega}\left(\varepsilon^{-2}\right)$ lower bound for the $\ell_{p}$ subspace sketch problem, when $d=\Theta(\log(1/\varepsilon))$. We then extend this to an $\widetilde{\Omega}\left(\varepsilon^{-2}d\right)$ lower bound for general $d$ via a simple padding argument. We first show how to prove a weaker $\widetilde{\Omega}\left(\varepsilon^{-1}\right)$ lower bound for the for-all version of the problem, and then show how to strengthen the argument to obtain both a stronger $\widetilde{\Omega}\left(\varepsilon^{-2}\right)$ lower bound and in the weaker original version of the problem (the “for-each” model, where we only need to be correct on a fixed query $x$ with constant probability).

Note that the condition that $d=\Theta(\log(1/\varepsilon))$ is crucial for our proof. As shown in Section 11, when $d=2$, there is actually an $\widetilde{O}(\varepsilon^{-1})$ upper bound, and thus our $\widetilde{\Omega}(\varepsilon^{-2})$ lower bound does not hold universally for all values of $d$. It is thus crucial that we look at a larger value of $d$, and we show that $d=\Theta(\log(1/\varepsilon))$ suffices.

To prove our bit lower bounds for the $\ell_{1}$ subspace sketch problem, we shall encode random bits in the matrix $A$ such that having a $(1+\varepsilon)$-approximation to $\|Ax\|_{1}$ will allow us to recover, in the for-each case, some specific random bit, and in the for-all case, all the random bits using different choices of $x$. A standard information-theoretic argument then implies that the lower bound for the subspace sketch problem is proportional to the number of random bits we can recover.

Warmup: An $\widetilde{\Omega}\left(\varepsilon^{-1}\right)$ Lower Bound for the For-All Version.

In our hard instance, we let $d=\Theta(\log(1/\varepsilon))$ be such that $n=2^{d}=\widetilde{\Theta}(1/\varepsilon)$. Form a matrix $A\in\mathbb{R}^{n\times d}$ by including all vectors $i\in\{-1,1\}^{d}$ as its rows and then scaling the $i$-th row by a nonnegative scalar $r_{i}\leq\operatorname{poly}(d)$. We can think of $r$ as a vector in $\mathbb{R}^{n}$ with $\|r\|_{\infty}\leq\operatorname{poly}(d)$. Now, we query $Q_{1}(i)$ for all vectors $i\in\{-1,1\}^{d}$. For an appropriate choice of $d=\Theta(\log(1/\varepsilon))$, for all $i\in\{-1,1\}^{d}$, we have

[TABLE]

Since $Q_{1}(i)$ is a $(1\pm\varepsilon)$-approximation to $\|Ai\|_{1}$, and $\|Ai\|_{1}$ is always an integer, we can recover the exact value of $\|Ai\|_{1}$ using $Q_{1}(i)$, for all $i\in\{-1,1\}^{d}$.

Now we define a matrix $M\in\mathbb{R}^{n\times n}$, where $M_{i,j}=|\langle i,j\rangle|$, where $i,j$ are interpreted as vectors in $\{-1,1\}^{d}$. A simple yet crucial observation is that, $\|Ai\|_{1}$ is exactly the $i$-th coordinate of $Mr$. Notice that this critically relies on the assumption that $r$ has nonnegative coordinates. Thus, the problem can be equivalently viewed as designing a vector $r\in\mathbb{R}^{n}$ with $\|r\|_{\infty}\leq\operatorname{poly}(d)$ and recovering $r$ from the vector $Mr$. At this point, a natural idea is to show that the matrix $M$ has a sufficiently large rank, say, $\operatorname{rank}(M)=\widetilde{\Omega}(\varepsilon^{-1})$, and carefully design $r$ to show an $\Omega(\operatorname{rank}(M))=\widetilde{\Omega}(\varepsilon^{-1})$ lower bound.

Fourier analysis on the hypercube shows that the eigenvectors of $M$ are the rows of the normalized Hadamard matrix, while the eigenvalues of $M$ are the Fourier coefficients associated with the function $g(s)=|d-2w_{H}(s)|$, where $w_{H}(s)$ is the Hamming weight of a vector $s\in\mathbb{F}_{2}^{d}$. Considering all vectors of Hamming weight $d/2$ in $\mathbb{F}_{2}^{d}$ and their associated Fourier coefficients, we arrive at the conclusion that there are at least $\binom{d}{d/2}$ eigenvalues of $M$ with absolute value

[TABLE]

which can be shown to be at least $\Omega(2^{d/2}/\operatorname{poly}(d))$. The formal argument is given in Section 3.1. Hence $\operatorname{rank}(M)\geq\binom{d}{d/2}=\Omega(2^{d}/\operatorname{poly}(d))$. Without loss of generality we assume the $\operatorname{rank}(M)\times\operatorname{rank}(M)$ upper-left block of $A$ is non-singular.

Now an $\widetilde{\Omega}(1/\varepsilon)$ lower bound follows readily. Set $r$ so that

[TABLE]

where $\{s_{i}\}_{i=1}^{\operatorname{rank}(M)}$ is a set of i.i.d. Bernoulli random variables. Since the exact value of $Mr$ is known and the $\operatorname{rank}(M)\times\operatorname{rank}(M)$ upper-left block of $A$ is non-singular, one can recover the values of $\{s_{i}\}_{i=1}^{\operatorname{rank}(M)}$ by solving a linear system, which implies an $\Omega(\operatorname{rank}(M))=\widetilde{\Omega}(\varepsilon^{-1})$ lower bound.

Before proceeding, let us first review why our argument fails for $p=2$. For the $\ell_{p}$-norm, the Fourier coefficients associated with the vectors of Hamming weight $d/2$ on the Boolean cube are

[TABLE]

Therefore this sum vanishes if and only if $p$ is an even integer, in which case $\operatorname{rank}(A)$ will no longer be $\Omega(2^{d}/\operatorname{poly}(d))$ and the lower bound argument will fail.

An $\widetilde{\Omega}\left(\varepsilon^{-2}\right)$ Lower Bound for the For-Each Version.

To strengthen this to an $\widetilde{\Omega}(\varepsilon^{-2})$ lower bound, it is tempting to increase $d$ so that $n=2^{d}=\widetilde{\Omega}(\varepsilon^{-2})$. In this case, however, we can no longer recover the exact value of $Mr$, since each entry of $Mr$ now has magnitude $\widetilde{\Theta}(\varepsilon^{-2})$ and the function $Q_{1}(\cdot)$ only gives a $(1\pm\varepsilon)$-approximation. We still obtain a noisy version of $Mr$, but with a $\widetilde{\Theta}(1/\varepsilon)$ additive error on each entry. One peculiarity of the model here is that if some entries of $r$ are negative, then $\|Ai\|_{1}=(M|r|)_{i}$ (cf. (3)), where $|r|$ denotes the vector formed by taking the absolute value of each coordinate of $r$, i.e., $\|Ai\|_{1}$ depends only on the absolute values of entries of $r$, which suggests that the constraint that each entry of $Mr$ has magnitude $\widetilde{\Theta}(1/\varepsilon^{2})$ with an additive error of $\widetilde{\Theta}(1/\varepsilon)$ is somehow intrinsic.

To illustrate our idea for overcoming the issue of large additive error, for the time being let us forget the actual form of $M$ previously defined in the argument for our $\widetilde{\Omega}\left(\varepsilon^{-1}\right)$ lower bound and consider instead a general $M\in\mathbb{R}^{n\times n}$ with orthogonal rows, each row having $\ell_{2}$ norm $\Omega(2^{d/2}/\operatorname{poly}(d))$. For now we also allow $r$ to contain negative entries such that $\|r\|_{\infty}\leq\operatorname{poly}(d)$, and pretend that the noisy version of $Mr$ has an $\widetilde{\Theta}(1/\varepsilon)$ additive error on each entry. Now, let

[TABLE]

where $\{s_{i}\}_{i=1}^{n}$ is a set of i.i.d. Rademacher random variables. By a standard concentration inequality, $\|r\|_{\infty}\leq\operatorname{poly}(d)$ holds with high probability (recall that $n=2^{d}$). Consider the vector $Mr$. Due to the orthogonality of the rows of $M$, the $i$-th coordinate of $Mr$ will be

[TABLE]

Provided that $\|M_{i}\|_{2}$ is larger than the additive error $\widetilde{\Theta}(1/\varepsilon)$, we can still recover $s_{i}$ by just looking at the sign of $\langle M_{i},r\rangle$. Thus, for an appropriate choice of $d$ such that $2^{d/2}/\operatorname{poly}(d)=\widetilde{\Omega}(1/\varepsilon)$, we can obtain an $\Omega(2^{d})=\widetilde{\Omega}(1/\varepsilon^{2})$ lower bound.

Now we return to the original $M$ with $M_{i,j}=|\langle i,j\rangle|$, whose rows are not necessarily orthogonal. The previous argument still goes through so long as we can identify a subset $\mathcal{R}\subseteq[n]=[2^{d}]$ of size $|\mathcal{R}|\geq\Omega(2^{d}/\operatorname{poly}(d))$ such that the rows $\{M_{i}\}_{i\in\mathcal{R}}$ are nearly orthogonal, meaning that the $\ell_{2}$ norm of the orthogonal projection of $M_{i}$ onto the subspace spanned by other rows $\{M_{j}\}_{j\in\mathcal{R}\setminus\{i\}}$ is much smaller than $\|M_{i}\|_{2}$.

To achieve this goal, we study the spectrum of $M$, and as far as we are aware, this is the first such study of spectral properties of this matrix. The Fourier argument mentioned above implies that at least $\Omega(2^{d}/\operatorname{poly}(d))$ eigenvalues of $A$ have the same absolute value $\Omega(2^{d/2}/\operatorname{poly}(d))$. If all other eigenvalues of $A$ were zero, then we could identify a set of $|\mathcal{R}|\geq\Omega(2^{d}/\operatorname{poly}(d))$ nearly orthogonal rows using rows of $A$ each with $\ell_{2}$ norm $\Omega(2^{d/2}/\operatorname{poly}(d))$, using a procedure similar to the standard Gram-Schmidt process. The full details can be found in Section 3.2. Although the other eigenvalues of $M$ are not all zero, we can simply ignore the associated eigenvectors since they are orthogonal to the set of nearly orthogonal rows we obtain above.

Lastly, recall that what we truly obtain is $M|r|$ rather than $Mr$ unless $r\geq 0$. To fix this, note that $\|r\|_{\infty}\leq\operatorname{poly}(d)$ with high probability, and so we can just shift each coordinate of $r$ by a fixed amount of $\operatorname{poly}(d)$ to ensure that all entries of $r$ are positive. We can still obtain $\langle M_{i},r\rangle$ with an additive error $\widetilde{\Theta}(1/\varepsilon)$, since the amount of the shift is fixed and bounded by $\operatorname{poly}(d)$.

Notice that the above argument in fact holds even for the for-each version of the subspace sketch problem. By querying the $i$-th vector on the Boolean cube for some $i\in\mathcal{R}$, we are able to recover the sign of $s_{i}$ with constant probability. Given this, a standard information-theoretic argument shows that our lower bound holds for the for-each version of the problem.

The formal analysis given in Section 3.3 is a careful combination of all the ideas mentioned above.

Applications: $M$-estimators and Projective Clustering Coresets.

Our general strategy for proving lower bounds for $M$-estimators is to relate one $M$-estimator, for which we want to prove a lower bound, to another $M$-estimator for which a lower bound is easy to derive. For the $L_{1}$-$L_{2}$ estimator, the Huber estimator and the Fair estimator, when $|t|$ is sufficiently large, $\phi(t)=(1\pm\varepsilon)|t|$ (up to rescaling of $t$ and the function value), and thus the lower bounds follow from those for the $\ell_{1}$ subspace sketch problem.

For the Cauchy estimator, we relate it to another estimator $\phi_{\text{aux}}(t)=\ln|x|\cdot\mathbf{1}_{\{|x|\geq 1\}}$. In Section 8, we show that our Fourier analytic arguments also work for $\phi_{\text{aux}}(t)$. Since for sufficiently large $t$, the Cauchy estimator satisfies $\phi(t)=(1\pm\varepsilon)\phi_{\text{aux}}(t)$ (up to rescaling of $t$ and the function value), a lower bound for the Cauchy estimator follows.

To prove lower bounds on coresets for projective clustering, the main observation is that when $k=1$ and $j=d-1$, by choosing the query subspace to be the orthogonal complement of a vector $z$, the projection cost is just $\sum_{x\in X}\phi(\langle x,z\rangle)$, and thus we can invoke our lower bounds for the subspace sketch problem. We use a coding argument to handle general $k$. In Lemma 9.1, we show there exists a set $S=\{(s_{1},t_{1}),(s_{2},t_{2}),\ldots,(s_{k},t_{k})\}$, where $s_{i},t_{i}\in\mathbb{R}^{O(\log k)}$, $\langle s_{i},t_{i}\rangle=0$ and $\langle s_{i},t_{j}\rangle$ is arbitrarily large when $i\neq j$. Now for $k$ copies of the hard instance of the subspace sketch problem, we add $s_{i}$ as a prefix to all data points in the $i$-th hard instance, and set the query subspace to be the orthogonal complement of a vector $z$, to which we add $t_{i}$ as a prefix. Now, the data points in the $i$-th hard instance will always choose the $i$-th center in the optimal solution, since otherwise an arbitrarily large cost will incur. Thus, we can solve $k$ independent copies of the subspace sketch problem, and the desired lower bound follows.

In the rest of the section, we shall illustrate our techniques for proving lower bounds that depend on $p$ for the $\ell_{p}$ subspace sketch problem. These lower bounds hold even when $\varepsilon$ is a constant. We again resort to information theory, trying to recover, using $Q_{p}$ queries, the entire matrix $A$ among a collection $\mathcal{S}$ of matrices. The lower bound is then $\Omega(\log|\mathcal{S}|)$ bits.

An $\widetilde{\Omega}(d^{p/2})$ Lower Bound for the For-Each Version.

Our approach for proving the $\widetilde{\Omega}(d^{p/2})$ lower bound is based on the following crucial observation: consider a uniformly random matrix $A\in\{-1,1\}^{\Theta(d^{p/2})\times d}$ and a uniformly random vector $x\in\{-1,1\}^{d}$. Then $\operatorname{\mathbb{E}}\|Ax\|_{p}^{p}=O(d^{p})$, whereas for each row $A_{i}\in\mathbb{R}^{d}$ of $A$, interpreted as a column vector, $\|AA_{i}\|_{p}^{p}\geq d^{p}$. Intuitively, the lower bound comes from the fact that one can recover the whole matrix $A$ by querying all Boolean vectors $x\in\{-1,1\}^{d}$ using the function $Q_{p}(\cdot)$, since if $x$ is a row of $A$, then $\|Ax\|_{p}^{p}$ would be slightly larger than its typical value, by adjusting constants.

To implement this idea, one can generate a set of almost orthogonal vectors $S\subseteq\mathbb{R}^{d}$ and require that all rows of $A$ come from $S$. A simple probabilistic argument shows that one can construct a set of $|S|=d^{p}$ vectors such that for any distinct $s,t\in S$, $\left|\langle s,t\rangle\right|\leq O(\sqrt{d\log d})$555The $O(\sqrt{\log d})$ factor can be removed using more sophisticated constructions based on coding theory (see Lemma 4.1).. If we form the matrix $A$ using $n=\widetilde{\Omega}(d^{p/2})$ vectors from $S$ as its rows, then for any vector $t$ that is not a row of $A$,

[TABLE]

for some appropriate choice of $n$. Thus, by querying $Q_{p}(s)$ for all vectors $s\in S$, one can recover the whole matrix $A$, even when $\varepsilon$ is a constant. By a standard information-theoretic argument, this leads to a lower bound of $\Omega\left(\log\binom{d^{p}}{\widetilde{\Omega}(d^{p/2})}\right)=\widetilde{\Omega}(d^{p/2})$ bits. Furthermore, one only needs to query $|S|=d^{p}$ vectors, which means the lower bound in fact holds for the for-each version of the $\ell_{p}$ subspace sketch problem, by a standard repetition argument and losing a $\log d$ factor in the lower bound.

An $\Omega(d^{\max\{p/2,1\}+1})$ Lower Bound for the For-All Version.

In order to obtain the nearly optimal $\Omega(d^{\max\{p/2,1\}+1})$ lower bound for the for-all version, we must abandon the constraint that all rows of the $A$ matrix come from a set $S$ of $\operatorname{poly}(d)$ vectors. Our plan is still to construct a large set of matrices $\mathcal{S}\subseteq\{+1,-1\}^{\Theta(d^{p/2})\times d}$, and show that for any distinct matrices $S,T\in\mathcal{S}$, it is possible to distinguish them using the function $Q_{p}(\cdot)$, thus proving an $\Omega(\log|\mathcal{S}|)$ lower bound. The new observation is that, to distinguish two matrices $S,T\in\mathcal{S}$, it suffices to have a single row of $T$, say $T_{i}$, such that $\|ST_{i}\|_{p}^{p}\ll d^{p}$. Again using the probabilistic method, we show the existence of such a set $\mathcal{S}$ with size $\exp\left(\Omega(d^{p/2+1})\right)$, which implies an $\Omega(\log|\mathcal{S}|)=\Omega(d^{p/2+1})$ lower bound.

Our main technical tool is Talagrand’s concentration inequality, which shows that for any $p\geq 2$ and vector $x\in\{-1,1\}^{d}$, for a matrix $A\in\mathbb{R}^{\Theta(d^{p/2})\times d}$ with i.i.d. Rademacher entries, $\|Ax\|_{p}=\Theta(d)$ with probability $1-\exp(-\Omega(d))$. This implies that for two random matrices $S,T\in\mathbb{R}^{\Theta(d^{p/2})\times d}$ with i.i.d. Rademacher entries, the probability that there exists some row $T_{i}$ of $T$ such that $\|ST_{i}\|_{p}^{p}\ll d^{p}$ is at least $1-\exp\left(\Omega(d^{p/2+1})\right)$, since the $\Theta(d^{p/2})$ rows of $T$ are independent. By a probabilistic argument, the existence of the set $\mathcal{S}$ follows. The formal analysis is given in Section 4.2.1.

The above argument fails to give an $\Omega(d^{2})$ lower bound when $p<2$. However, for any $p<2$, since $\ell_{2}^{n}$ embeds into $\ell_{p}^{m}$ with $m=O_{p}(n)$ and a constant distortion, we can directly reduce the case of $p<2$ to the case of $p=2$. The formal analysis can be found in Section 4.2.2. Combining these two results yields the $\Omega(d^{\max\{p/2,1\}+1})$ lower bound.

2 Preliminaries

For a vector $x\in\mathbb{R}^{n}$, we use $\|x\|_{p}$ to denote its $\ell_{p}$-norm, i.e., $\|x\|_{p}=\left(\sum_{i=1}^{n}|x_{i}|^{p}\right)^{1/p}$. When $p<1$, it is not a norm but it is still a well-defined quantity and we call it the $\ell_{p}$-norm for convenience. When $p=0$, $\|x\|_{0}$ is defined to be the number of nonzero coordinates of $x$.

For two vectors $x,y\in\mathbb{R}^{n}$, we use $\operatorname{proj}_{y}x\in\mathbb{R}^{n}$ to denote the orthogonal projection of $x$ onto $y$. For a matrix $A\in\mathbb{R}^{n\times d}$, we use $A_{i}\in\mathbb{R}^{d}$ to denote its $i$-th row, treated as a column vector. We use $\|A\|_{2}$ to denote its spectral norm, i.e., $\|A\|_{2}=\sup_{\|x\|_{2}=1}\|Ax\|_{2}$, and $\|A\|_{F}$ to denote its Frobenius norm, i.e., \|A\|_{F}=\big{(}\sum_{i=1}^{n}\sum_{j=1}^{d}A_{ij}^{2}\big{)}^{1/2}.

Suppose that $A\in\mathbb{R}^{m\times n}$ has singular values $\sigma_{1}\geq\sigma_{2}\geq\cdots\geq\sigma_{r}\geq 0$, where $r=\min\{m,n\}$. It holds that \sigma_{1}=\|A\|_{2}\leq\|A\|_{F}=\big{(}\sum_{i=1}^{r}\sigma_{i}^{2}\big{)}^{1/2}. The condition number of $A$ is defined to be

[TABLE]

Theorem 2.1** (Eckart–Young–Mirsky Theorem).**

Suppose that $A\in\mathbb{R}^{m\times n}$ has singular values $\sigma_{1}\geq\sigma_{2}\geq\cdots\geq\sigma_{r}>0$, where $\operatorname{rank}(A)=r\leq\min\{m,n\}$. For any matrix $B\in\mathbb{R}^{m\times n}$ such that $\operatorname{rank}(B)\leq k\leq r$, it holds that

[TABLE]

Below we list a handful of concentration inequalities which will be useful in our arguments.

Lemma 2.2** (Hoeffding’s inequality, [6, p34]).**

Let $s_{1},\dots,s_{n}$ be i.i.d. Rademacher random variables and $a_{1},\dots,a_{n}$ be real numbers. Then

[TABLE]

Lemma 2.3** (Khintchine’s inequality, [6, p145]).**

Let $s_{1},\dots,s_{n}$ be i.i.d. Rademacher random variables and $a_{1},\dots,a_{n}$ be real numbers. There exist absolute constants $A,B>0$ such that

[TABLE]

Lemma 2.4** (Talagrand’s inequality, [6, p204]).**

Let $X=(X_{1},\dots,X_{n})$ be a random vector with independent coordinates taking values in $[-1,1]$. Let $f:[-1,1]^{n}\to\mathbb{R}$ be a convex $1$-Lipschitz function. It holds for all $t\geq 0$ that

[TABLE]

Lemma 2.5** (Gaussian concentration, [39, p105]).**

Let $p\geq 1$ be a constant. Consider a random vector $X\sim N(0,I_{n})$ and a non-negative $1$-Lipschitz function $f:(\mathbb{R}^{n},\|\cdot\|_{2})\to\mathbb{R}$, then

[TABLE]

where $c=c(p)>0$ is a constant that depends only on $p$.

Lemma 2.6** (Extreme singular values, [39, p91]).**

Let $A$ be an $N\times n$ matrix with i.i.d. Rademacher entries. Let $\sigma_{\min}(A)$ and $\sigma_{\max}(A)$ be the smallest and largest singular values of $A$. Then for every $t\geq 0$, with probability at least $1-2\exp(ct^{2})$, it holds that

[TABLE]

where $C,c>0$ are absolute constants.

Lemma 2.7**.**

Let $U_{1},\dots,U_{k}\in\mathbb{R}^{n}$ be orthonormal vectors and $s_{1},\dots,s_{k}$ be independent Rademacher random variables. It holds that

[TABLE]

Proof.

Let $Z=\sum_{i=1}^{k}s_{i}U_{i}$, then

[TABLE]

Since $\left\{U_{i}\right\}$ is a set of orthonormal vectors, we have that

[TABLE]

It follows from Hoeffding’s inequality (Lemma 2.2) that for each $j\in[k]$,

[TABLE]

The claimed inequality follows by taking a union bound over all $j\in[k]$. ∎

We also need a result concerning uniform approximation of smooth functions by polynomials. Let $P_{n}$ denote the space of polynomials of degree at most $n$. For a given function $f\in C[a,b]$, the best degree-$n$ approximation error $E_{n}(f;[a,b])$ is defined to be

[TABLE]

where the $\|\cdot\|_{\infty}$ norm is taken over $[a,b]$. The following bound on approximation error is a classical result.

Lemma 2.8** ([33, p23]).**

Let $f(x)$ have a $k$-th derivative on $[-1,1]$. If $n>k$,

[TABLE]

where $\omega_{k}$ is the modulus of continuity of $f^{(k)}$, defined as

[TABLE]

3 An $\widetilde{\Omega}\left(\varepsilon^{-2}\right)$ Lower Bound

To prove the space lower bound of the data structure $Q_{p}$, we appeal to information theory. We shall encode random bits in $A$ such that if for each $x$, $Q_{p}(x)$ approximates $\|Ax\|_{p}^{p}$ (or $\|Ax\|_{p}$ when $p=0$) up to a $1\pm\varepsilon$ factor with probability at least $0.9$, we can recover from $Q_{p}(x)$ some random bit (depending on $x$) with at least constant probability. A standard information-theoretic argument implies a lower bound on the size of $Q_{p}$ which is proportional to the number of random bits we can recover.

For each $p\geq 0$, we define a family of matrices $M^{(p)}=\{M^{(d,p)}\}_{d=1}^{\infty}$, where $M^{(d,p)}$ is a $2^{d}\times 2^{d}$ matrix with entries defined as

[TABLE]

where $i$ and $j$ are interpreted as vectors on the Boolean cube $\{-1,1\}^{d}$. We assume $0^{0}=0$ throughout the paper.

3.1 Spectrum of Matrices $M$

Lemma 3.1**.**

For any $d\geq 1$, $M^{(d,p)}$ can be rewritten as $H^{(d)}\Lambda^{(d,p)}(H^{(d)})^{T}$ in its spectral decomposition form, where $\Lambda^{(d,p)}$ is a $2^{d}\times 2^{d}$ diagonal matrix, and $H^{(d)}$ is a $2^{d}\times 2^{d}$ normalized Hadamard matrix.

Proof.

Let $T$ be the natural isomorphism from the multiplicative group $\{-1,1\}^{d}$ to the additive group $\mathbb{F}_{2}^{d}$. Then $|\langle i,j\rangle|^{p}=g(Ti+Tj)$ for some function $g$ defined on $\mathbb{F}_{2}^{d}$. It can be computed (see [25, Lemma 5]) that the singular values of a matrix with entries $g(Ti+Tj)$ are the absolute values of the Fourier coefficients of $g$. In our particular case, the singular values of $M^{(d,p)}$ are

[TABLE]

Furthermore, the proof of that lemma shows that $H^{(d)}$ in the spectral decomposition is given by

[TABLE]

which implies that $H^{(d)}$ is a normalized Hadamard matrix. ∎

Lemma 3.2**.**

When $d$ is even, there are at least $\binom{d}{d/2}$ entries in $\Lambda^{(d,p)}$ with absolute value

[TABLE]

Proof.

We shall use the notation in the proof of Lemma 3.1. Consider the Fourier coefficients $\hat{g}(s)$ for $s\in\mathbb{F}_{2}^{d}$ with Hamming weight $d/2$, which is the same for all $\binom{d}{d/2}$ such $s$’s. Note that

[TABLE]

By comparing the coefficients of $x^{i}$ on both sides of the identity $(1+x)^{d/2}(1-x)^{d/2}=(1-x^{2})^{d/2}$, we see that

[TABLE]

Hence

[TABLE]

Finally, observe that, for $x,y\in\{+1,-1\}^{d}$, we have $(d-\langle x,y\rangle)/2=d_{H}(x,y)=w_{H}(Tx+Ty)$, where $d_{H}(x,y)$ denotes the Hamming distance between $x$ and $y$ and $w_{H}(s)$ denotes the Hamming weight of $s\in\mathbb{F}_{2}^{d}$. Hence $g(i)=|d-2i|^{p}$ and the conclusion follows. ∎

Let $N^{(d)}$ be the multiplicity of the singular value $\Lambda^{(d,p)}_{0}$ of $M^{(d,p)}$. We know from the preceding lemma that $N^{(d)}\geq\binom{d}{d/2}$. By permuting the columns of $H^{(d)}$, we may assume the absolute value of the first $N^{(d)}$ diagonal entries of $\Lambda^{(d,p)}$ are all equal to $\Lambda^{(d,p)}_{0}$, i.e.,

[TABLE]

The following lemma is critical in lower bounding $\Lambda^{(d,p)}_{0}$. We found the result in a post on math.stackexchange.com [1] but could not find it in any published literature and so we reproduce the proof in full from [1], with small corrections regarding convergence of integrals.

Lemma 3.3**.**

It holds for all complex $p$ satisfying $0<\operatorname{Re}p<2n$ that

[TABLE]

Proof.

By the binomial theorem,

[TABLE]

Splitting the sum at $k=-1$ and $k=1$, we have

[TABLE]

Plugging in $z=\exp{(2it)}$ yields

[TABLE]

Plug (5) into the integral on the right-hand side of (4) and introduce a regularizer $\exp(-st)$ ($s>0$) under the integral sign:

[TABLE]

One can compute that

[TABLE]

It follows that

[TABLE]

It is easy to verify that the integral on the left-hand side is analytic whenever the integral converges. Analytic continuation permits $p$ to be extended to $\{p:-1<\operatorname{Re}p<2n\}\setminus\mathbb{Z}$. Now, for $p$ such that $0<\operatorname{Re}p<2n$ and $p\notin\mathbb{Z}$, let $s\to 0^{+}$ on both sides. It is also easy to verify that we can take the limit $s\to 0^{+}$ under the integral sign, hence

[TABLE]

Invoking the reflection identity (see, e.g. [4, p9])

[TABLE]

we obtain that

[TABLE]

Finally, analytic continuation extends $p$ to the integers in $(0,2n)$. ∎

As an immediate corollary of Lemma 3.3, we have:

Corollary 3.4**.**

Suppose that $d\in 8\mathbb{Z}$. There exists an absolute constant $c>0$ such that

[TABLE]

Proof.

Letting $2n=d/2$ and $k=n-i/2$, the summation in Lemma 3.2 becomes

[TABLE]

Since (cf. [14, p511])

[TABLE]

where $C>0$ is an absolute constant, we have that

[TABLE]

Notice that $h(p)=\Gamma(p+1)\zeta(p+1)/(\pi/2)^{p+1}$ is a positive continuous function on $(0,\infty)$ and $h(p)\to\infty$ as $p\to\infty$ and $p\to 0^{+}$, it must hold that $\inf_{p>0}h(p)>0$. The conclusion follows. ∎

3.2 Orthogonalizing Rows

Suppose we are given a matrix $\Pi\in\mathbb{R}^{n\times n}$ in its spectral decomposition form $\Pi=H\Sigma H^{T}$, where

[TABLE]

and $H$ is the normalized Hadamard matrix. The goal of this section is to identify a set of orthogonal vectors, using rows of $\Pi$.

Lemma 3.5**.**

Each row of $\Pi$ has the same $\ell_{2}$ norm $\|\Pi_{i}\|_{2}=\sigma\sqrt{r/n}$.

Proof.

[TABLE]

The lemma follows since all entries in $H$ have absolute value $1/\sqrt{n}$, and the $r$ non-zero entries on the diagonal of $\Sigma$ have absolute value $\sigma$. ∎

To identify a set of orthogonal vectors using the rows of $\Pi$, we run a procedure similar to the standard Gram-Schmidt process.

Lemma 3.6**.**

There is a set $\mathcal{R}\subseteq[n]$ with size $|\mathcal{R}|=r/100$ such that for each $i\in\mathcal{R}$, $\Pi_{i}$ can be written as

[TABLE]

where $\left\{R_{i}\right\}_{i\in\mathcal{R}}$ is a set of orthogonal vectors, and $P_{i}$ is the orthogonal projection of $\Pi_{i}$ onto the subspace spanned by $\left\{R_{j}\right\}_{j\in\mathcal{R}\setminus\{i\}}$. Furthermore, for each $i\in\mathcal{R}$, $\|R_{i}\|_{2}^{2}\geq 99/100\|\Pi_{i}\|_{2}^{2}=99/100\cdot\sigma^{2}r/n$.

Proof.

We show how to construct such a set $\mathcal{R}$. Suppose that we have found a set $\mathcal{R}$ with size strictly less than $r/100$ with $\Pi_{i}=R_{i}+P_{i}$ satisfying the stated constraints. We shall show how to increase the size of $\mathcal{R}$ by one.

Let $\Pi=S+Q$. Here, for each $i\in[n]$ we have $\Pi_{i}=S_{i}+Q_{i}$, where $Q_{i}$ is the orthogonal projection of $\Pi_{i}$ onto the subspace spanned by $\left\{R_{j}\right\}_{j\in\mathcal{R}}$ and $S_{i}=\Pi_{i}-Q_{i}$. Notice that for all $j\in\mathcal{R}$ we have $Q_{j}=\Pi_{j}$ and $S_{j}=0$. Since $\|\Pi\|_{F}^{2}=r\sigma^{2}$ and $\operatorname{rank}(Q)\leq|\mathcal{R}|$, by Theorem 2.1 we have

[TABLE]

Thus, by averaging, there exists $i\notin\mathcal{R}$ such that $\|S_{i}\|_{2}^{2}>99/100\|\Pi_{i}\|_{2}^{2}$. We add $i$ into $\mathcal{R}$ and set $R_{i}$ in (8) to be $S_{i}$ and $P_{i}$ to be $Q_{i}$. It is easy to verify that the stated constraints still hold. We continue this process inductively until $|\mathcal{R}|=r/100$. ∎

Lemma 3.7**.**

Suppose that $e\in\mathbb{R}^{n}$ satisfies $\|e\|_{\infty}\leq 0.1\sigma\sqrt{r/n}$. Let $x\in\mathbb{R}^{n}$ be a random vector defined as

[TABLE]

where $\{s_{i}\}_{i\in\mathcal{R}}$ is a set of i.i.d. Rademacher random variables. Here the set $\mathcal{R}$ and the orthogonal vectors $\{R_{i}\}_{i\in\mathcal{R}}$ are as defined in Lemma 3.6. For each $i\in\mathcal{R}$, it holds that

[TABLE]

Proof.

For each $i\in\mathcal{R}$, we have

[TABLE]

We first analyze the second term.

[TABLE]

By Markov’s inequality, with probability at least $4/5$, we have $\left|\langle P_{i},x\rangle\right|\leq\|\Pi_{i}\|_{2}/2$.

Recall that $\|R_{i}\|_{2}\geq 99/100\|\Pi_{i}\|_{2}$ (Lemma 3.6) and $\|\Pi_{i}\|_{2}=\sigma\sqrt{r/n}$ (Lemma 3.5). It happens with probability at least $4/5$ that $|e_{i}|+\left|\langle P_{i},x\rangle\right|<\left|\langle R_{i},x\rangle\right|$, in which case we have $\operatorname{sign}\left((\Pi x+e)_{i}\right)=\operatorname{sign}(s_{i})$. ∎

3.3 Space Lower Bound on $Q_{p}$

In this section, we describe a reduction from the subspace sketch problem to the $\mathsf{INDEX}$ problem, a classical problem in communication complexity. We shall rephrase the problem in the context of a data structure. The $\mathsf{INDEX}$ data structure stores an input string $s\in\{-1,1\}^{n}$ and supports a query function, which receives an input $i\in[n]$ and outputs $s_{i}\in\{-1,1\}$ which is the $i$-th bit of the underlying string. To prove the lower bound for the subspace sketch problem, we need the following lower bound for the distributional $\mathsf{INDEX}$ problem.

Lemma 3.8** ([29]).**

In the $\mathsf{INDEX}$ problem, suppose that the underlying string $s$ is drawn uniformly from $\{-1,1\}^{n}$ and the input $i$ of the query function is drawn uniformly from $[n]$. Any (randomized) data structure for $\mathsf{INDEX}$ that succeeds with probability at least $2/3$ requires $\Omega(n)$ bits of space, where the randomness is taken over both the randomness in the data structure and the randomness of $s$ and $i$.

Throughout the reduction, $d$ is a fixed parameter with value to be determined later. For the matrix $M^{(d,p)}$, we consider its spectrum-truncated version

[TABLE]

Lemma 3.9**.**

Each row of $\tilde{M}^{(d,p)}$ is orthogonal to all eigenvectors associated with eigenvalues other than $\Lambda^{(d,p)}_{1},\Lambda^{(d,p)}_{2},\ldots,\Lambda^{(d,p)}_{N^{(d)}}$.

Proof.

Let $v_{1},\dots,v_{2^{d}}$ be the columns of $H^{(d)}$. Then $\tilde{M}^{(d,p)}=\sum_{i=1}^{N^{(d)}}\Lambda_{i}^{(d,p)}v_{i}v_{i}^{T}$. Let $w$ be an eigenvector corresponding to another eigenvalue. Then

[TABLE]

since $v_{i}$ and $w$ are orthogonal as they are associated with distinct eigenvalues. ∎

Now we invoke Lemma 3.6 on the matrix $\tilde{M}^{(d,p)}$ and obtain a set $\mathcal{R}\subseteq[2^{d}]$ and a set of orthogonal vectors $\{R_{i}\}_{i\in\mathcal{R}}$. We shall encode $|\mathcal{R}|$ random bits in $A$ and show how to recover them.

Let

[TABLE]

By Lemma 2.7, with probability $1-\exp(-\Omega(d))$, it holds that $\|x\|_{\infty}\leq 3\sqrt{d}$. We condition on $\|x\|_{\infty}\leq 3\sqrt{d}$ in the rest the proof, since we can include the alternative case $\|x\|_{\infty}>3\sqrt{d}$ in the overall failure probability.

Next we define a vector $y\in\mathbb{R}^{2^{d}}$ to be $y_{i}=(x_{i}+\Delta^{(d)})^{1/p}$, where $\Delta^{(d)}=5\sqrt{d}$ is a constant that depends only on $d$. Clearly, it holds for all $i\in[2^{d}]$ that $2\sqrt{d}\leq y_{i}^{p}\leq 8\sqrt{d}$. Round each entry of $y$ to its nearest integer multiple of $\delta=1/(p(8\sqrt{d})^{1-1/p}2^{d})$, obtaining $\tilde{y}$. A simple calculation using the mean-value theorem shows that for all $i\in[2^{d}]$,

[TABLE]

Finally we construct the matrix $A\in\mathbb{R}^{2^{d}\times d}$ to be used in the $\ell_{p}$ subspace sketch problem. The $j$-th row of $A$ is the $j$-th vector of $\{-1,1\}^{d}$, scaled by $\tilde{y}_{j}$.

Lemma 3.10**.**

The matrix $A$ constructed above for the $\ell_{p}$ subspace sketch problem satisfies $\kappa(A)\leq C$ for some constant $C$ that depends on $p$ only.

Proof.

Let $B$ be the $2^{d}\times d$ matrix whose rows are all vectors in $\{-1,1\}^{d}$. Then,

[TABLE]

where $s_{1},\dots,s_{d}$ is a Rademacher sequence. It follows from Khintchine’s inequality that

[TABLE]

for some constants $C_{1},C_{2}$. Notice that the rows of $A$ are rescaled rows of $B$ with the scaling factors in $[(2\sqrt{d})^{1/p},(8\sqrt{d})^{1/p}]$. Hence $\kappa(A)\leq C$ for some constant $C$ that depends on $p$ only. ∎

The recovery algorithm is simple. The vector to be used for querying the data structure is the $i$-th vector on the Boolean cube $\{-1,1\}^{d}$, where $i\in\mathcal{R}$. Given $Q_{p}(i)$, we guess the sign of $s_{i}$ to be just the sign of $Q_{p}(i)-\langle M^{(d,p)}_{i},\Delta^{(d)}\cdot\mathbf{1}\rangle$. Next we prove the correctness of the recovery algorithm.

The guarantee of the subspace sketch problem states that, with probability at least $0.9$, it holds simultaneously for all $i\in\{-1,1\}^{d}$ that

[TABLE]

We condition on this event in the remaining part of the analysis.

First we notice that for any $i\in\{-1,1\}^{d}$,

[TABLE]

Next we give an upper bound on the value of $\|Ai\|_{p}^{p}$, for all $i\in\{-1,1\}^{d}$.

Lemma 3.11**.**

For each $i\in\{-1,1\}^{d}$, the matrix $A$ constructed for the $\ell_{p}$ subspace sketch problem satisfies

[TABLE]

Proof.

Each term in the summation (12) is upper bounded by $(8d^{1.5})^{p}$, which implies the stated lemma. ∎

Combining the preceding lemma with the query guarantee (11), the preceding lemma implies that, it holds for all $i\in\mathbb{R}^{2^{d}}$ that

[TABLE]

On the other hand, by (9) and (12),

[TABLE]

Thus by the triangle inequality,

[TABLE]

Notice that $x$ is a linear combination of rows of $\tilde{M}^{(d,p)}$. By Lemma 3.9,

[TABLE]

By Lemma 3.7, if

[TABLE]

then with probability $4/5$, $(Q_{p}(i)-\langle M^{(d,p)}_{i},\Delta^{(d)}\cdot\mathbf{1}\rangle)$ has the same sign as $\left(\tilde{M}^{(d,p)}x\right)_{i}$, in which case we recover the correct sign. By Lemma 3.8, the size of $Q_{p}$ is lower bounded by $\Omega(|\mathcal{R}|)$.

Now for each $\varepsilon>0$ and $p\in(0,\infty)\setminus 2\mathbb{Z}$, by Lemma 3.2, (13) can be satisfied by setting

[TABLE]

which implies a space complexity lower bound of

[TABLE]

bits.

Formally, we have proved the following theorem.

Theorem 3.12**.**

Let $p\in(0,\infty)\setminus 2\mathbb{Z}$. There exist constants $C\in(0,1]$ and $\varepsilon_{0}>0$ that depend only on $p$ such that the following holds. Let $d_{0}=2\log_{2}(C/(\varepsilon\operatorname{polylog}(1/\varepsilon))$. For any $\varepsilon\in(0,\varepsilon_{0})$, $d\geq d_{0}$ and $n\geq 2^{d_{0}}$, any data structure for the $\ell_{p}$ subspace sketch requires $\Omega\left(\frac{1}{\varepsilon^{2}\cdot\operatorname{polylog}(1/\varepsilon)}\right)$ bits. The lower bound holds even when $\kappa(A)\leq K$ for some constant $K$ that only depends on $p$.

We note that the $\operatorname{polylog}(1/\varepsilon)$ factors in the definition of $d_{0}$ and the bit lower bound may not have the same exponent.

Next we strengthen the lower bound to $\widetilde{\Omega}(d/\varepsilon^{2})$ bits.

Corollary 3.13**.**

Under the assumptions of $C$, $\varepsilon_{0}$, $d$ in Theorem 3.12 and the assumption that $n=\Omega\left(\frac{d}{\varepsilon^{2}\cdot\operatorname{polylog}(1/\varepsilon)}\right)$, any data structure for the $\ell_{p}$ subspace sketch problem requires $\Omega\left(\frac{d}{\varepsilon^{2}\cdot\operatorname{polylog}(1/\varepsilon)}\right)$ bits. The $\operatorname{polylog}(1/\varepsilon)$ factors in the two $\Omega$-notations may not have the same exponent.

Proof.

Let $A^{\prime}\in\mathbb{R}^{n^{\prime}\times d^{\prime}}$ be the hard instance matrix for Theorem 3.12, where $d^{\prime}=2\log_{2}(C/(\varepsilon\operatorname{polylog}(1/\varepsilon))$ and $n^{\prime}=2^{d^{\prime}}$. We construct a block diagonal matrix $A$ with $b=d/d^{\prime}$ blocks, each being an independent copy of $A$’, so that $A$ has $d$ columns. The number of rows in $A$ is $bn^{\prime}=\Omega\left(\frac{d}{\varepsilon^{2}\operatorname{polylog}(1/\varepsilon)}\right)$. In this case, the $\ell_{p}$ sketch problem on $A^{\prime}$ requires a data structure of $\widetilde{\Omega}(b/\varepsilon^{2})=\Omega\left(\frac{d}{\varepsilon^{2}\cdot\operatorname{polylog}(1/\varepsilon)}\right)$ bits, since we are now solving the INDEX problem with $\widetilde{\Omega}(b\cdot 1/\varepsilon^{2})$ random bits. ∎

The corollary above is also true for $p=0$.

Corollary 3.14**.**

Under the assumptions of $C$, $\varepsilon_{0}$, $d$ and $n$ in Corollary 3.13, any data structure for the $\ell_{0}$ subspace sketch problem requires $\Omega\left(\frac{d}{\varepsilon^{2}\cdot\operatorname{polylog}(1/\varepsilon)}\right)$ bits.

Proof.

The matrix $M^{(d,0)}$ is defined as $(M^{(d,0)})_{i,j}=\mathbf{1}_{\{\langle i,j\rangle\neq 0\}}$. Note that each row of $M^{(d,0)}$ has the same number of $1$s; let $W_{d}$ denote this number. Observe that Lemma 3.4 continues to hold because we have by symmetry

[TABLE]

for some absolute constant $c>0$. Let $y_{j}=x_{i}+\Delta^{(d)}$, where $x_{i}$ and $\Delta^{(d)}$ are as defined before. In the construction of $A$, replicate $\tilde{y}_{j}$ times (rounded to an integer multiple of $\delta=2^{-d}$) the $j$-th vector of $\{-1,1\}^{d}$. Our guess of the sign $s_{i}$ is then the sign of $\delta Q_{0}(i)-\Delta^{(d)}W_{d}$. Similar to the procedure above, we have that

[TABLE]

and

[TABLE]

And therefore it suffices to have

[TABLE]

which holds when $2^{d/2}=1/(\varepsilon/\operatorname{polylog}(1/\varepsilon))$ as before. Therefore the analogue of Theorem 3.12 holds and so does the analogue of Corollary 3.13. ∎

Remark 3.15**.**

The condition that $p\notin 2\mathbb{Z}^{+}$ is necessary for the lower bound. When $p\in 2\mathbb{Z}^{+}$, it is possible to achieve $\varepsilon=0$ with $O(d^{p}\log(nd))$ words. Recall that a $d$-dimensional subspace of $\ell_{p}$ space can be isometrically embedded into $\ell_{p}^{r}$ with $r=\binom{d+p-1}{p}-1$ [23]. In general the data structure does not necessarily correspond to a linear map and can be of any form. Indeed, there is a much simpler data structure as follows, based on ideas in [35]. For each $x\in\mathbb{R}^{d}$, let $y_{x}\in\mathbb{R}^{d}$ be defined as $(y_{x})_{i}=((Ax)_{i})^{p/2}$, then $\|y_{x}\|_{2}^{2}=\|Ax\|_{p}^{p}$. Observe that each coordinate $(y_{x})_{i}$ is a polynomial of $d^{p/2}$ terms in $x_{1},\dots,x_{d}$. Form an $n\times d^{p/2}$ matrix $B$, where the $i$-th row consists of the coefficients in the polynomial corresponding to $(y_{x})_{i}$. The data structure stores $B^{T}B$. To answer the query $Q_{p}(x)$, one first calculates from $x$ a $d^{p/2}$-dimensional vector $x^{\prime}$ whose coordinates are all possible monomials of total degree $p/2$. Note that $Bx^{\prime}=y_{x}$. Hence one can just answer $Q_{p}(x)=(x^{\prime})^{T}B^{T}Bx^{\prime}=\|Bx^{\prime}\|_{2}^{2}=\|Ax\|_{p}^{p}$ without error. This $Q_{p}$ does not give an isometric embedding but is much simpler than known isometric embeddings, and the space complexity is $O(d^{p}\log(nd))$ bits.

4 Lower Bounds for $p>2$

4.1 Lower Bounds for the Subspace Sketch Problem for $p>2$

In this section, we prove a lower bound on the $\ell_{p}$ subspace sketch problem, in the case that $\varepsilon$ is a constant and $p\geq 2$. We need the following result from coding theory.

Lemma 4.1** ([32]).**

For any $p\geq 1$ and $d=2^{k}-1$ for some integer $k$, there exist a set $S\subset\{-1,1\}^{d}$ and a constant $C_{p}$ depending only on $p$ which satisfy

1. (i)

$|S|=d^{p}$; 2. (ii)

For any $s,t\in S$ such that $s\neq t$, $|\langle s,t\rangle|\leq C_{p}\sqrt{d}$.

Lemma 4.2**.**

For any $p\geq 1,C\geq 1$ and $d=2^{k}-1$ for an integer $k$, there exist a set $S\subset\{-1,1\}^{d}$ with size $|S|=d^{p}$, a set $\mathcal{M}\subset\mathbb{R}^{R\times d}$ for some $R$ and a constant $C_{p}$ depending only on $p$ which satisfy

1. (i)

For any $M_{1},M_{2}\in\mathcal{M}$ such that $M_{1}\neq M_{2}$, there exists $x\in S$ such that $\|Mx\|_{p}<d/C$ and $\|Mx\|_{p}\geq d$. 2. (ii)

$|\mathcal{M}|\geq\exp\left(d^{p/2}/(C_{p}C^{p})\right)$.

Proof.

Set $R=d^{p/2}/(C_{p}C^{p})$. Then $R\leq d^{p}/e$. We set $\mathcal{M}$ to be the set of $R\times d$ matrices whose rows are all possible combinations of $R$ distinct vectors in $S$, where $S$ is the set constructed in Lemma 4.1. Clearly, $|\mathcal{M}|={d^{p}\choose R}\geq e^{R}$. Furthermore, consider two different $M_{1},M_{2}\in M$. There exists an $x\in S$ which is a row of $M_{1}$ but not a row of $M_{2}$. Thus, $\|M_{1}x\|_{p}\geq d$ and

[TABLE]

Theorem 4.3**.**

Solving the $\ell_{p}$ subspace sketch problem requires $\widetilde{\Omega}(d^{p/2})$ bits when $0<\varepsilon<1$ and $p\geq 2$ are constants and $n=\Omega(d^{p/2})$.

Proof.

We first prove a lower bound for randomized data structures for the $\ell_{p}$ subspace sketch problem with failure probability $d^{-p}/100$. Let $\mathcal{M}\subset\mathbb{R}^{R\times d}$ and $S\subset\{-1,1\}^{d}$ be as constructed in Lemma 4.2. Choose a matrix $M$ from $\mathcal{M}$ uniformly at random. Since for each $x\in\{-1,1\}^{d}$, with probability at least $1-d^{-p}/100$,

[TABLE]

by a union bound, with probability at least $0.99$, (14) holds simultaneously for all $x\in S$. It follows from Lemma 4.2(i) that by querying $\|Mx\|_{p}$ for all $x\in S$, one can distinguish all different $M\in\mathcal{M}$. A standard information-theoretic argument leads to a lower bound of $\Omega(\log|\mathcal{M}|)=\Omega(d^{p/2})$.

For randomized data structures for the $\ell_{p}$ subspace sketch problem with constant failure probability, a standard repetition argument implies that the failure probability can be reduced to $d^{-p}/100$ using $O(\log d)$ independent repetitions. Therefore a lower bound of $\widetilde{\Omega}(d^{p/2})$ bits follows. ∎

Remark 4.4**.**

The lower bound in Theorem 4.3 is nearly optimal. To obtain an $\ell_{p}$ subspace sketch with constant $\varepsilon$ and $\widetilde{O}(d^{p/2})$ bits, one can first apply Lewis weights sampling [13] to reduce the size of $A$ to $\widetilde{O}(d^{p/2})\times d$, and then apply the embedding in [15] to further reduce the number of rows of $A$ to $\widetilde{O}(d^{(p/2)\cdot(1-2/p)})=\widetilde{O}(d^{p/2-1})$. Therefore the data structure takes $\widetilde{O}(d^{p/2})$ bits to store.

4.2 Lower Bounds for the For-All Version

In this section, we prove a lower bound on the for-all version of the $\ell_{p}$ subspace sketch problem for the case of $p\geq 2$ and constant $\varepsilon$. In the for-all version of the $\ell_{p}$ subspace sketch problem, the data structure $Q_{p}$ is required to, with probability at least $0.9$, satisfy $Q_{p}(x)=(1\pm\varepsilon)\|Ax\|_{p}$ simultaneously for all $x\in\mathbb{R}^{d}$.

4.2.1 Lower Bound for $p\geq 2$

Throughout this section we assume that $p\geq 2$ is a constant.

Let $N=c_{p}d^{p/2}$ in this section, where $c_{p}>0$ is a constant that depends only on $p$. Denote the unit ball in $\ell_{p}^{n}$ by $B_{p}^{n}$. For each $x\in B_{2}^{n}$, we define a function $f_{x}:\mathbb{R}^{N\times d}\to\mathbb{R}$ by

[TABLE]

Lemma 4.5**.**

The function $f_{x}(\cdot)$ satisfies the following properties:

1. (i)

$\operatorname{\mathbb{E}}[f_{x}(A)]\leq Cc_{p}^{1/p}\sqrt{p}\sqrt{d}$, where entries of $A$ are i.i.d. Rademacher random variables and $C$ is a constant that depends only on $c_{p}$; 2. (ii)

$f_{x}(\cdot)$* is $1$-Lipschitz with respect to the Frobenius norm;* 3. (iii)

$f_{x}(\cdot)$* is a convex function.*

Proof.

By Khintchine’s inequality, $(\operatorname{\mathbb{E}}|(Ax)_{i}|^{p})^{1/p}\leq C\sqrt{p}\|x\|_{2}=C\sqrt{p}$, where $C$ is an absolute constant. It follows that $\operatorname{\mathbb{E}}\|Ax\|_{p}^{p}\leq N(C\sqrt{p})^{p}$ and by Jensen’s inequality, $\operatorname{\mathbb{E}}\|Ax\|_{p}\leq(\operatorname{\mathbb{E}}\|Ax\|_{p}^{p})^{1/p}\leq N^{1/p}C\sqrt{p}=Cc_{p}^{1/p}\sqrt{p}\sqrt{d}$, which implies (i). To prove (ii), note that

[TABLE]

(iii) is a simple consequence of the convexity of the $\ell_{p}$ norm. ∎

The following lemma is a direct application of Talagrand’s concentration inequality (Lemma 2.4) with Lemma 4.5.

Lemma 4.6**.**

Let $A\in\mathbb{R}^{N\times d}$ and $x\in\mathbb{R}^{d}$ have i.i.d. Rademacher random variables. It holds that

[TABLE]

where $C$ is an absolute constant and $c_{p}$ is a constant depending only on $p$.

Proof.

Let $\hat{x}=x/\sqrt{d}$. We have $\|\hat{x}\|_{2}=1$. By Lemma 4.5 and Lemma 2.4, we have

[TABLE]

Since $f_{x}(A)=\sqrt{d}f_{\hat{x}}(A)$, we have

[TABLE]

which implies the stated lemma. ∎

Lemma 4.7**.**

There exists a multiset $\mathcal{S}\subseteq\{+1,-1\}^{N\times d}$ such that

1. (i)

$|\mathcal{S}|\geq\exp(c_{1}Nd)$; 2. (ii)

For any $S,T\in\mathcal{S}$ such that $S\neq T$, there exists $i\in[N]$, such that $\|ST_{i}\|_{p}\leq Cc_{p}^{1/p}\sqrt{p}d$; 3. (iii)

When $p>2$, for any $S\in\mathcal{S}$, $\kappa(S)\leq 2$.

Proof.

We first define a set of bad matrices $\mathsf{Bad}\subseteq\{+1,-1\}^{N\times d}$ to be

[TABLE]

where $x\in\{+1,-1\}^{d}$ is an i.i.d. Rademacher vector and $C_{p},c$ are the same constants in Lemma 4.6. It follows from Lemma 4.6 that

[TABLE]

since otherwise

[TABLE]

Let the multiset $\mathcal{T}\subseteq\{+1,-1\}^{N\times d}$ of size $|\mathcal{T}|=\exp(c_{2}Nd)$ consist of independent uniform samples of matrices in $\{+1,-1\}^{N\times d}$. We define three events as follows.

• •

$\mathcal{E}_{1}$: $|\mathcal{T}\setminus\mathsf{Bad}|\geq|\mathcal{T}|/3$;

• •

$\mathcal{E}_{2}$: For each $S\in\mathcal{T}\setminus\mathsf{Bad}$ and each $T\in\mathcal{T}\setminus\{S\}$, there exists some $i\in[N]$ such that $\|ST_{i}\|_{p}\leq Cpd$.

• •

$\mathcal{E}_{3}$: There are at least $(5/6)|\mathcal{T}|$ matrices $T\in\mathcal{T}$ such that $\kappa(T)\leq 2$.

We analyze the probability of each event below.

First, notice that $\operatorname{\mathbb{E}}|\mathcal{T}\cap\mathsf{Bad}|\leq|\mathcal{T}|/3$. Thus, by Markov’s inequality we have $\Pr(|\mathcal{T}\cap\mathsf{Bad}|\geq 2|\mathcal{T}|/3)\leq 1/2$, which implies $\Pr(\mathcal{E}_{1}^{c})\leq 1/2$.

Next, consider a fixed matrix $S\in\{+1,-1\}^{N\times d}\setminus\mathsf{Bad}$. For a random matrix $T\in\{+1,-1\}^{N\times d}$ whose entries are i.i.d. Rademacher random variables, for each row $T_{i}$ of $T$, by definition of $\mathsf{Bad}$, we have

[TABLE]

Since the rows of $T$ are independent,

[TABLE]

Choosing appropriate constants for $C$ and $c$ (and thus $c^{\prime}$) allows for a union bound over all pairs $S\in\mathcal{T}\setminus\mathsf{Bad}$ and $T\in\mathcal{T}\setminus\{S\}$, and we have $\Pr(\mathcal{E}_{2}^{c})\leq 1/3$.

Last, for the condition number, recall the classical result that for a random matrix $T$ of i.i.d. Rademacher entries, it holds with probability $\geq 1-\exp(-c_{3}d)$ that $s_{\min}(T)\geq\sqrt{N}-c_{4}\sqrt{d}$ and $s_{\min}(T)\geq\sqrt{N}+c_{4}\sqrt{d}$, which implies that $\kappa(T)\leq(\sqrt{N}+c_{4}\sqrt{d})/(\sqrt{N}-c_{4}\sqrt{d})\leq 2$ when $d$ is sufficiently large. Letting $\mathcal{T}_{1}=\{T\in|\mathcal{T}|:\kappa(T)>2\}$, we have $\operatorname{\mathbb{E}}|\mathcal{T}_{1}|\leq e^{-c_{3}d}|\mathcal{T}|$. Thus by a Markov bound, $\Pr\{|\mathcal{T}_{1}|\geq 6e^{-c_{3}d}|\mathcal{T}|\}\leq 1/10$, and thus $\Pr(\mathcal{E}_{3}^{c})\leq 1/10$.

Since $\Pr(\mathcal{E}_{1}^{c})+\Pr(\mathcal{E}_{2}^{c})+\Pr(\mathcal{E}_{3}^{c})<1$, there exists a set $\mathcal{T}$ for which all $\mathcal{E}_{1}$, $\mathcal{E}_{2}$, $\mathcal{E}_{3}$ hold. Taking $\mathcal{S}$ to be the well-conditioned matrices in $\mathcal{T}\setminus\mathsf{Bad}$, we see that $\mathcal{S}$ satisfies conditions (i)–(iii). ∎

Theorem 4.8**.**

The for-all version of the $\ell_{p}$ subspace sketch problem requires $\Omega((d/p)^{p/2}\cdot d)$ bits to solve when $p\geq 2$ and $\varepsilon<1$ are constants and $n=\Omega(d^{p/2})$. The lower bound holds even when $\kappa(A)\leq 2$ if $p>2$, and all entries in $A$ are in $\{+1,-1\}$.

Proof.

Choose a matrix $A$ uniformly at random from the set $\mathcal{S}$ in Lemma 4.7. Suppose that $Q_{p}:\mathbb{R}^{d}\to\mathbb{R}$ satisfies

[TABLE]

For any row $A_{i}\in\mathbb{R}^{d}$ of $A$, interpreted as a column vector, $\|AA_{i}\|_{p}\geq d$, whereas for any $B\in\mathcal{S}\setminus\{A\}$, there exists a row $A_{i}$ of $A$ such that $\|BA_{i}\|_{p}\leq Cc_{p}^{1/p}\sqrt{p}d<d/3$, provided that $c_{p}$ (and thus $C$) is small enough. Thus, by appropriate choice of the constants in Lemma 4.7, we can use $Q_{p}$ to determine which matrix $A\in\mathcal{S}$ has been chosen. By Property (ii) of the set $\mathcal{S}$, it must hold that all elements of $\mathcal{S}$ are distinct from each other. It then follows from a standard information-theoretic argument that the size of the data structure for the $\ell_{p}$ sketch problem is lower bounded by $\Omega(\log|\mathcal{S}|)=\Omega(Nd)=\Omega((d/p)^{p/2}\cdot d)$. ∎

4.2.2 Lower Bound for $1\leq p\leq 2$

The lower bound for $1\leq p<2$ follows from the lower bound for $p=2$ by embedding $\ell_{p}$ into $\ell_{2}$. It is known that $\ell_{2}^{n}$ $K$-embeds into $\ell_{p}^{m}$ for some $m\leq cn$, where $c=c(p)$ and $K=K(p)$ are constants that depend only on $p$. Furthermore, the embedding $T:\ell_{2}^{n}\to\ell_{p}^{m}$ can be realized using a rescaled matrix of i.i.d. Rademacher entries (with high probability). See [27, Section 2.5] for a proof for $p=1$, which can be generalized easily to a general $p$. Thus, one can reduce the for-all version of the $\ell_{2}$ subspace sketch problem to the for-all version of the $\ell_{p}$ subspace sketch with $1\leq p\leq 2$. Thus the lower bound of $\Omega(d^{2})$ also holds when $1\leq p\leq 2$.

5 Linear Embeddings

In this section, our goal is to show that isomorphic embeddings into low-dimensional spaces induce solutions to the subspace sketch problem. Therefore a lower bound on the subspace sketch problem implies a lower bound on the embedding dimension.

Theorem 5.1**.**

Let $p,q\geq 1$, $\varepsilon>0$ and $A\in\mathbb{R}^{N\times d}$ with full column rank. Let $E\subseteq\ell_{p}^{N}$ be the column space of $A$ and suppose that $T:E\to\ell_{q}^{n}$ is a $(1+\varepsilon)$-isomorphic embedding. Then there exists a data structure for the for-all version of the $\ell_{p}$ subspace sketch problem on $A$ with approximation ratio $1\pm 6p\varepsilon$ and $O(nd\log(N^{|1/p-1/2|}dn\kappa(A)/\varepsilon))$ bits.

Proof.

Without loss of generality, we may assume that $\frac{1}{\kappa(A)}\|x\|_{2}\leq\|Ax\|_{2}\leq\|x\|_{2}$. Let $B\in\mathbb{R}^{n\times d}$ be such that $B=TA$. Then $\|Ax\|_{p}^{p}\leq\|Bx\|_{q}^{p}\leq(1+\varepsilon)^{p}\|Ax\|_{p}^{p}$. Round each entry of $B$ to an integer multiple of $\delta=\varepsilon/(D_{1}n^{1/q}d^{1/2}\kappa(A))$, where $D_{1}=\max\{1,N^{1/2-1/p}\}$, obtaining $\tilde{B}$. First we claim that the rounding causes only a minor loss,

[TABLE]

Indeed, write $B=\tilde{B}+\Delta B$, where each entry of $\Delta B$ is bounded by $\delta$. Then

[TABLE]

This proves (15) and so

[TABLE]

which implies that the matrix $\tilde{B}$ can be used to solve the $\ell_{p}$ subspace sketch problem on $A$ with approximation ratio $1\pm 6p\varepsilon$. Since

[TABLE]

where $D_{2}=\max\{1,N^{1/p-1/2}\}$, each entry of $B$ is at most $eD_{2}$. Hence, after rounding, each entry of $\tilde{B}$ can be described in $O(\log(D_{2}/\delta))=O(\log(dnD\kappa(A)/\varepsilon))$ bits, where $D=D_{1}D_{2}=N^{|1/2-1/p|}$. The matrix $\tilde{B}$ can be described in $O(nd\log(D_{2}/\delta))$ bits. The value of $\delta$ can be described in $O(\log(1/\delta))$ bits, which is dominated by the complexity for describing $\tilde{B}$. Therefore the size of the data structure is at most $O(nd\log(D_{2}/\delta))=O(nd\log(Ddn\kappa(A)/\varepsilon))$ bits. ∎

The dimension lower bound for linear embeddings now follows as a corollary from combining the preceding theorem with Theorem 3.12, where we choose $d=C\log(1/\varepsilon)$ and note that $N=O(1/\varepsilon^{2})$ and $\kappa(A)=O(1)$ in our hard instance.

Corollary 5.2**.**

Let $p\in[1,\infty)\setminus 2\mathbb{Z}$ and suppose that $d\geq C\log(1/\varepsilon)$. It holds that

[TABLE]

where $c_{p}>0$ is a constant that depends only on $p$ and $C>0$ is an absolute constant.

Remark 5.3**.**

It is not clear how much the assumption $d\geq C\log\frac{1}{\varepsilon}$ can be weakened. The best known results for $p=1$ are as follows [23].

[TABLE]

which is substantially better than $1/(\varepsilon^{2}\operatorname{polylog}(1/\varepsilon))$ for constant $d$. In a similar lower bound [7]

[TABLE]

the constant $c(d)\approx e^{-c^{\prime}d\ln d}$, so the lower bound is nontrivial for $d$ up to $O\left(\log\frac{1}{\varepsilon}/\log\log\frac{1}{\varepsilon}\right)$. Since $N_{1}(d,\varepsilon)$ is increasing, optimizing $d$ w.r.t. $\varepsilon$ yields that N_{1}(d,\varepsilon)=\Omega\big{(}\varepsilon^{-2}\exp(-c^{\prime\prime}\sqrt{\ln(1/\varepsilon)\ln\ln(1/\varepsilon)})\big{)} for all d=\Omega\big{(}\sqrt{\ln(1/\varepsilon)}\big{)}. Our result improves the lower bound to $\varepsilon^{-2}/\operatorname{polylog}(1/\varepsilon)$ for larger $d$ and, more importantly, works for general $p\geq 1$ that is not an even integer.

Remark 5.4**.**

In the case of $p>2$, it is an immediate corollary from Theorem 4.8 that $N_{p}(d,\varepsilon)=\Omega(d^{p/2}/\log d)$, which recovers the known (and nearly tight) lower bound up to a logarithmic factor.

6 Sampling-based Embeddings

Our goal in this section is to prove the following lower bound.

Theorem 6.1**.**

Let $p\geq 1$ and $p\notin 2\mathbb{Z}$. Suppose that $Q_{p}(x)=\|TAx\|_{p}^{p}$ solves the for-all version of the $\ell_{p}$ subspace sketch problem on $A$ and $\varepsilon$ for some $T\in\mathbb{R}^{m\times n}$ such that each row of $T$ contains exactly one non-zero element. Then it must hold that $m\geq c_{p}d/(\varepsilon^{2}\operatorname{polylog}(1/\varepsilon))$, provided that $d\geq C\log(1/\varepsilon)$, where $c_{p}>0$ is a constant that depends only on $p$ and $C>0$ is an absolute constant.

Proof.

Let $A$ be the hard instance matrix for the $\widetilde{\Omega}(1/\varepsilon^{2})$ lower bound in Corollary 5.2. Recall that $A$ is a diagonal matrix with $k=\Theta(d/\log(1/\varepsilon))$ diagonal blocks. Each block has dimension $2^{s}\times s$ with $2^{s/2}=1/\varepsilon^{1-o(1)}$. Furthermore, each block can be written as $DB$, where $D$ is a $2^{s}\times 2^{s}$ diagonal matrix and $B$ is the $2^{s}\times s$ matrix whose rows are all vectors in $\{-1,1\}^{s}$, and each entry in $D$ has magnitude in $[(2\sqrt{s})^{1/p},(8\sqrt{s})^{1/p}]$. The matrix $B$ can be described using $O(s2^{s})$ bits and the matrix $D$ using $O(2^{s}\log s)$ bits. Without loss of generality we may assume that each nonzero entry of $T$ is an integer multiple of $\varepsilon^{1/p}$, since the loss of rounding, by the triangle inequality, is at most $\varepsilon\|Ax\|_{p}^{p}$. Next, we shall bound the number of bits needed to describe $T$.

Letting $x=e_{j}$ be a canonical basis vector,

[TABLE]

On the other hand,

[TABLE]

where we used (10) for the last inequality. It follows from the AM-GM inequality that

[TABLE]

that is, $T$ can be described using $O(ms)$ bits, provided that $m\geq k$. Therefore $TA$ can be described in $O(s2^{s}+2^{s}\log s+ms)=O((m+1/\varepsilon^{2})\log(1/\varepsilon))$ bits. Combining with the lower bound of $\Omega(d/(\varepsilon^{2}\operatorname{poly}\log(1/\varepsilon))$ bits, we see that $m=\Omega(d/(\varepsilon^{2}\operatorname{polylog}(1/\varepsilon)))$.

A similar argument shows that when $m<k$, the matrix $T$ can be described in $O(d)$ bits, which leads to a contradiction to the lower bound. Hence it must hold that $m\geq k$ and, as we proved above, $m=\Omega(d/(\varepsilon^{2}\operatorname{polylog}(1/\varepsilon)))$. ∎

The lower bound for the (for-each) $\ell_{p}$ subspace sketch problem loses further a factor of $\log d$.

Corollary 6.2**.**

Let $p\geq 1$ and $p\notin 2\mathbb{Z}$. Suppose that $Q_{p}(x)=\|TAx\|_{p}^{p}$ solves the (for-each) version of the $\ell_{p}$ subspace sketch problem on $A$ and $\varepsilon$ for some $T\in\mathbb{R}^{m\times n}$ such that each row of $T$ contains exactly one non-zero element. Then it must hold that $m\geq c_{p}d/(\varepsilon^{2}\cdot\log d\cdot\operatorname{polylog}(1/\varepsilon))$, provided that $d\geq C\log(1/\varepsilon)$, where $c_{p}>0$ is a constant that depends only on $p$ and $C>0$ is an absolute constant.

Proof.

Observe that we used the approximation to $\|Ae_{i}\|_{p}^{p}$ for each canonical basis vector $e_{i}$ in the proof of Theorem 6.1, which holds with a constant probability if we make $O(\log d)$ independent copies of the (randomized) data structure. This incurs a further loss of a $\log d$ factor in the lower bound. ∎

7 Oblivious Sketches

An oblivious subspace embedding for $d$-dimensional subspaces $E$ in $\ell_{p}^{n}$ is a distribution on linear maps $T:\ell_{p}^{n}\to\ell_{p}^{m}$ such that it holds for any $d$-dimensional subspace $E\subseteq\ell_{p}^{n}$ that

[TABLE]

More generally, an oblivious sketch is a distribution on linear maps $T:\ell_{p}^{n}\to\mathbb{R}^{m}$, accompanied by a recovery algorithm $\mathcal{A}$, such that it holds for any $d$-dimensional subspace $E\subseteq\ell_{p}^{n}$ that

[TABLE]

It is clear that an oblivious embedding is a special case of an oblivious sketch, where $\mathcal{A}(Tx)=\|Tx\|_{p}$.

In this section we shall show that when $1\leq p<2$, any oblivious sketch requires $m=\widetilde{\Omega}(d/\varepsilon^{2})$.

Before proving the lower bound, let us prepare some concentration results. We use $\mathbb{S}^{n-1}$ to denote the unit sphere in $(\mathbb{R}^{n},\|\cdot\|_{2})$. First, observe that the norm function $x\mapsto\|x\|_{p}$ is a Lipschitz function of Lipschitz constant $\max\{1,n^{1/p-1/2}\}$. Also note that $(\operatorname{\mathbb{E}}_{g\sim N(0,I_{n})}\|g\|_{p}^{p})^{1/p}=\beta_{p}n^{1/p}$, where $\beta_{p}=(\operatorname{\mathbb{E}}_{g\sim N(0,1)}|g|^{p})^{1/p}$. Standard Gaussian concentration (Lemma 2.5) leads to the following:

Lemma 7.1**.**

Let $p\geq 1$ be a constant and $g\sim N(0,I_{n})$. It holds with probability at least $1-\exp(-c\varepsilon^{2}n^{\min\{1,2/p\}})$ that $(1-\varepsilon)\beta_{p}n^{1/p}\leq\|g\|_{p}\leq(1+\varepsilon)\beta_{p}n^{1/p}$, where $c=c(p)>0$ is a constant that depends only on $p$.

Suppose that $G$ is an $n\times d$ Gaussian random matrix of i.i.d. $N(0,1)$ entries. Observe that for a fixed $x\in\mathbb{S}^{d-1}$, $Gx\sim N(0,I_{n})$. A typical $\varepsilon$-net argument on $\mathbb{S}^{d-1}$ allows us to conclude the following lemma. We remark that this gives Dvoretzky’s Theorem for $\ell_{p}$ spaces.

Lemma 7.2**.**

Let $1\leq p<2$ be a constant and $G$ be an $n\times d$ Gaussian random matrix. There exist constants $C=C(p)>0$ and $c=c(p)>0$ such that whenever $n\geq Cd\log(1/\varepsilon)/\varepsilon^{2}$, it holds $\Pr\left\{(1-\varepsilon)\beta_{p}n^{1/p}\leq\|Gx\|_{p}\leq(1+\varepsilon)\beta_{p}n^{1/p},\ \forall x\in\mathbb{S}^{d-1}\right\}\geq 1-2\exp(-c\varepsilon^{2}n)$.

Now, consider two distributions on $n\times d$ matrices, where $n=\Theta(d\varepsilon^{-2}\log(1/\varepsilon))$. The first distribution $\mathcal{L}_{1}$ is just the distribution of a Gaussian random matrix $G$ of i.i.d. $N(0,1)$ entries, and the second distribution $\mathcal{L}_{2}$ is the distribution of $G+\sigma uv^{T}$, where $G$ is the Gaussian random matrix of i.i.d. $N(0,1)$ entries, $u\sim N(0,I_{n})$ and $v\sim N(0,I_{d})$ and $\sigma=\alpha\sqrt{\varepsilon/d}$ for some constant $\alpha$ to be determined later, and $G$, $u$ and $v$ are independent.

Theorem 7.3**.**

Let $1\leq p<2$ be a constant. Suppose that $S\in\mathbb{R}^{m\times n}$ is an oblivious sketch for $d$-dimensional subspaces in $\ell_{p}^{n}$, where $n=\Theta(d\varepsilon^{-2}\log(1/\varepsilon))$. It must hold that $m\geq cd/\varepsilon^{2}$, where $c=c(p)>0$ is a constant depending only on $p$.

Proof.

It follows from the preceding lemma that, if $A\sim\mathcal{L}_{1}$, we have $\sup_{x\in\mathbb{S}^{d-1}}\|Ax\|_{p}\leq(1+\varepsilon)\beta_{p}n^{1/p}$ with probability at least $0.999$ with an appropriate choice of constant in the $\Theta$-notation of $n$. Next we consider the supremum of $\|Ax\|_{p}$ when $A\sim\mathcal{L}_{2}$. Observe that

[TABLE]

Since $v\sim N(0,I_{d})$, the direction $v/\|v\|_{2}\sim\operatorname{Unif}(\mathbb{S}^{d-1})$ and the magnitude $\|v\|_{2}$ are independent, and by rotational invariance of the Gaussian distribution, $Gx\sim N(0,I_{d})$ for any $x\in\mathbb{S}^{d-1}$. Hence

[TABLE]

where $t$ follows the distribution of $\|v\|_{2}$ and $u_{1},u_{2}$ are independent $N(0,I_{n})$ vectors. Applying the preceding two lemmata, we see that with probability at least $0.998$, it holds that $t\geq 0.99\sqrt{d}$ and $\|u\|_{p}\geq(1-\varepsilon)\beta_{p}n^{1/p}$. Therefore, when $A\sim\mathcal{L}_{2}$, with probability at least $0.998$, we have $\sup_{x\in\mathbb{S}^{d-1}}\|Ax\|_{p}\geq\sqrt{1+0.99^{2}\alpha^{2}\varepsilon}(1-\varepsilon)\beta_{p}n^{1/p}\geq(1+4\varepsilon)\beta_{p}n^{1/p}$, for an appropriate choice of $\alpha$.

Therefore with the corresponding recovery algorithm $\mathcal{A}$,

[TABLE]

which implies that the linear sketch $S$ can be used to distinguish $\mathcal{L}_{1}$ from $\mathcal{L}_{2}$ by evaluating $\sup_{x\in\mathbb{S}^{d-1}}\mathcal{A}(SAx)$. It then follows from [26, Theorem 4] that the size of the sketch $md\geq c/\sigma^{4}=c^{\prime}d^{2}/\varepsilon^{2}$ for some absolute constants $c,c^{\prime}>0$, and thus $m\geq c^{\prime}d/\varepsilon^{2}$. ∎

8 Lower Bounds for $M$-estimators

The main theorem of this section is the following.

Theorem 8.1**.**

Suppose there exist $\alpha,\lambda>0$ and $p\in(0,\infty)\setminus 2\mathbb{Z}$ such that $\phi(t/\lambda)\sim\alpha|t|^{p}$ as $t\to\infty$ or $t\to 0$. Then the subspace sketch problem for $\Phi(x)=\sum_{i=1}^{n}\phi(x_{i})$ requires $\Omega(d/(\varepsilon^{2}\operatorname{polylog}(1/\varepsilon)))$ bits when $d\geq C_{1}\log(1/\varepsilon)$ and $n\geq C_{2}d/(\varepsilon^{2}\operatorname{polylog}(1/\varepsilon))$ for some absolute constants $C_{1},C_{2}>0$.

Proof.

We reduce the problem to the $\ell_{p}$ subspace sketch problem. We prove the statement in the case of $t\to\infty$ below. The proof for the case of $t\to 0$ is similar.

For a given $\varepsilon>0$, there exists $M$ such that $(1-\varepsilon)\alpha|t|^{p}\leq\phi(t/\lambda)\leq(1+\varepsilon)\alpha|t|^{p}$ for all $|t|\geq M$. Let $A$ be our hard instance for the $\ell_{p}$ subspace sketch problem in Theorem 3.12. Then each row of $A$ is a $\{-1,1\}$-vector scaled by a factor of $\tilde{y}_{i}\geq\Delta$ for some \Delta=\Omega\big{(}\log^{1/(2p)}(1/\varepsilon)\big{)}. One can recover a random sign used in the construction of $A$ by querying $Ax$ for a $\{-1,1\}$-vector $x$. Therefore, if $(Ax)_{i}\neq 0$, it must hold that $|(Ax)_{i}|\geq\Delta$. This implies that there exists a scaling factor $\beta=M/\Delta$ such that $(1-\varepsilon)\alpha\|\beta Ax\|_{p}^{p}\leq\Phi(\lambda^{-1}\beta Ax)\leq(1+\varepsilon)\alpha\|\beta Ax\|_{p}^{p}$, that is, $\alpha^{-1}\beta^{-p}\Phi(\lambda^{-1}\beta Ax)$ is a $(1\pm\varepsilon)$-approximation to $\|Ax\|_{p}^{p}$ for $\{-1,1\}$-vectors $x$. The conclusion follows from Theorem 3.13 (which plants independent copies of hard instance $A$ in diagonal blocks) and a rescaling of $\varepsilon$. ∎

We have the following immediate corollary.

Corollary 8.2**.**

The subspace sketch problem for $\Phi$ requires $\Omega(d/(\varepsilon^{2}\operatorname{polylog}(1/\varepsilon)))$ bits when $d\geq C_{1}\log(1/\varepsilon)$ and $n\geq C_{2}d/(\varepsilon^{2}\operatorname{polylog}(1/\varepsilon))$ for some absolute constants $C_{1},C_{2}>0$ for the following functions $\phi$:

• •

($L_{1}$-$L_{2}$* estimator) $\phi(t)=2(\sqrt{1+t^{2}/2}-1)$;*

• •

(Huber estimator) $\phi(t)=t^{2}/(2\tau)\cdot\mathbf{1}_{\{|t|\leq\tau\}}+(|t|-\tau/2)\cdot\mathbf{1}_{\{|t|>\tau\}}$;

• •

(Fair estimator) $\phi(t)=\tau^{2}(|x|/\tau-\ln(1+|t|/\tau))$;

• •

(Tukey loss $p$-norm) $\phi(t)=|t|^{p}\cdot\mathbf{1}_{\{|t|\leq\tau\}}+\tau^{p}\cdot\mathbf{1}_{\{|t|>\tau\}}$.

Now we prove the $\Omega(d/(\varepsilon^{2}\operatorname{polylog}(1/\varepsilon)))$ lower bound for the subspace sketch problem for the Cauchy estimator $\phi(t)=(\tau^{2}/2)\ln(1+(t/\tau)^{2})$. First consider an auxiliary function $\phi_{\text{aux}}(t)=\ln|x|\cdot\mathbf{1}_{\{|x|\geq 1\}}$, for which we shall have also an $\Omega(d/(\varepsilon^{2}\operatorname{polylog}(1/\varepsilon))$ lower bound by following the approach in Section 3 with some changes we highlight below. Instead of $M_{i,j}^{(d,p)}=|\langle i,j\rangle|^{p}$, we shall define $M_{i,j}^{(d,p)}=\phi_{\text{aux}}(\langle i,j\rangle)$, and we proceed to define $N^{(d)}$ and $\Lambda_{0}^{(d,p)}$ in the same manner. The following lemma is similar to Corollary 3.4, showing that this new matrix $M^{(d,p)}$ also has large singular values. The proof is postponed to Section 8.1.

Lemma 8.3**.**

Suppose that $d\in 8\mathbb{Z}$. Then $\Lambda_{0}^{(d,p)}\geq c2^{d/2}/\sqrt{d}$ for some absolute constant $c>0$.

Therefore, the entire lower bound argument in Corollary 3.14 goes through. We can then conclude that the subspace sketch problem for $\Phi_{\text{aux}}(x)=\sum_{i=1}^{n}\phi_{\text{aux}}(x_{i})$ requires $\Omega(d/(\varepsilon^{2}\operatorname{polylog}(1/\varepsilon))$ bits. Now, for the Cauchy estimator $\phi(t)=(\tau^{2}/2)\ln(1+(t/\tau)^{2})$, note that $(1-\varepsilon)\tau^{2}\phi_{\text{aux}}(t)\leq\phi(\tau\cdot t)\leq(1+\varepsilon)\tau^{2}\phi_{\text{aux}}(t)$ for all sufficiently large $t$. It follows from a similar argument to the proof of Theorem 8.1 that the same lower bound continues to hold for the subspace sketch problem for the Cauchy estimator.

Corollary 8.4**.**

The subspace sketch problem for $\Phi$ requires $\Omega(d/(\varepsilon^{2}\operatorname{polylog}(1/\varepsilon)))$ bits for the Cauchy estimator $\phi(t)=(\tau^{2}/2)\ln(1+(t/\tau)^{2})$, when $d\geq C_{1}\log(1/\varepsilon)$ and $n\geq C_{2}d/(\varepsilon^{2}\operatorname{polylog}(1/\varepsilon))$ for some absolute constants $C_{1},C_{2}>0$.

8.1 Proof of Lemma 8.3

Differentiate both sides of (6) w.r.t $p$,

[TABLE]

Thus using the reflection identity (7),

[TABLE]

Letting $p\to 0^{+}$, we see that the middle term on the right-hand side vanishes, which implies

[TABLE]

Invoking Lemma 3.3, we obtain that

[TABLE]

We claim that the limit on the right-hand side at least $c/\sqrt{n}$ for some absolute constant $c>0$. Note that letting $p\to 0^{+}$ in Lemma 3.3 leads to

[TABLE]

and the left-hand side is

[TABLE]

thus

[TABLE]

Note the fact that $\Gamma^{\prime}(x)/\Gamma(x)=-1/x-\gamma+o(1)$ as $x\to 0^{+}$ (e.g., plugging $n=1$ into Eq. (1.2.15) in [4, p13]), where $\gamma=0.577\cdots$ is the Euler gamma constant. Thus the limit on the right-hand side of (16) is the same as

[TABLE]

and it suffices to show that

[TABLE]

for some constant c\in\big{(}0,\frac{4}{3}(\gamma+\ln 2)\big{)}. Since the limit above must exist, we can pick a sequence $p_{k}\to 0$ such that $e^{1/p_{k}}$ is a multiple of $\pi$. Hence we assume that $N=e^{1/p}/\pi$ is an integer below.

Now, split the integral into $[0,\pi]$ and $[\pi,\infty)$. Observe that

[TABLE]

because the integrands, viewed as functions of $(p,t)$, are bounded on $[0,1]\times[0,\pi]$, since $\sin^{2n}t\sim t^{2n}$ near $t=0$ and so $t=0$ is not a singularity. Furthermore,

[TABLE]

because the integrand is uniformly continuous on $[0,1]\times[0,\pi]$ and we can take the limit under the integral sign.

Now we deal with the integral on $[\pi,\infty)$. Following our approach in Corollary 3.4, we have that

[TABLE]

where

[TABLE]

and we used the fact that $(1-p\ln t)/t^{p+1}$ is nonnegative and decreasing when $\ln t\leq 1/p$.

The bracketed term on the rightmost side of (18) is

[TABLE]

The second term clearly tends to [math] as $p\to 0^{+}$, while the first term is equal to

[TABLE]

where $\zeta(p)=\sum_{n=1}^{\infty}1/n^{p}$ is the Riemann zeta function and we used the fact that $\zeta(1+p)=\frac{1}{p}+\gamma+f(p)$ for an analytic function $f$ on $\mathbb{C}$ with $f(0)=0$ (see, e.g. [4, p15]).

Therefore we conclude that the limit on the left-hand side in (17) is at least

[TABLE]

and it suffices to show that

[TABLE]

for some

[TABLE]

First observe that

[TABLE]

and

[TABLE]

Letting $\delta=\sqrt{(\ln n)/n}$, then $\sqrt{n}\sin^{2n}t\to 0$ as $n\to\infty$ on $t\in[1,\pi/2-\delta]$, and thus

[TABLE]

and we can now bound

[TABLE]

Therefore we conclude that we can choose $c_{1}$ to be

[TABLE]

as desired.

9 Lower Bounds on Coresets for Projective Clustering

We shall prove a lower bound of $\widetilde{\Omega}(kj/\varepsilon^{2})$ bits for coresets for projective clustering. First we need a lemma which provides codewords to encode the clustering information.

Lemma 9.1**.**

For any given integer $L\geq 1$ and even integer $D\geq 2$, there exists a set $S=\{(s_{1},t_{1}),(s_{2},t_{2}),\ldots,(s_{m},t_{m})\}$ of size $m\geq c2^{D}/\sqrt{D}$, where $s_{i},t_{i}\in\mathbb{R}^{D}$ and $c>0$ is an absolute constant, such that

• •

$\langle s_{i},t_{i}\rangle=0$;

• •

$\langle s_{i},t_{j}\rangle\geq L^{2}$* for $i\neq j$;*

• •

all entries of $s_{i}$ and $t_{i}$ are in $\{0,L\}$.

Proof.

We first consider the case $L=1$. Let $\{s_{i}\}$ be the set of all binary vectors with Hamming weight $D/2$, and $t_{i}=\mathbf{1}^{D}-s_{i}$, i.e., $t_{i}$ is the complement of $s_{i}$. Thus, $\langle s_{i},t_{i}\rangle=0$ by construction. For any $i\neq j$, since $s_{i}\neq s_{j}$, and both $s_{i}$ and $s_{j}$ have Hamming weight $D/2$, we have $\langle s_{i},t_{j}\rangle\geq 1$.

For a general $L$, we replace all entries of value $1$ in the construction above with $L$. ∎

In the rest of the section, we also use an $n\times d$ matrix to represent a point set of size $n$ in $\mathbb{R}^{d}$, where each row represents a point in $\mathbb{R}^{d}$.

Below we set up the framework of the hard instance for the projective subspace clustering problem. For a given $k$, choosing $D=O(\log k)$, we can obtain a set $S$ of size $k$ as guaranteed by Lemma 9.1. Suppose that $j\geq D+1$ and $d\geq j+1$. Without loss of generality we may assume that $d=j+1$, otherwise we just embed our hard instance in $\mathbb{R}^{j+1}$ into $\mathbb{R}^{d}$ by appending zero coordinates.

For a set $\mathcal{A}$ consisting of $k$ matrices $A^{(1)},A^{(2)},\ldots,A^{(k)}\in\mathbb{R}^{n\times(j+1-D)}$, we form a point set $X=X(\mathcal{A})\in\mathbb{R}^{nk\times d}$, whose rows are indexed by $(i,j)\in[k]\times[n]$ and defined as

[TABLE]

where $A^{(i)}_{j}$ denotes the $j$-th row of $A^{(i)}$.

Suppose that $y\in\mathbb{R}^{j+1-D}$. For each $i\in[k]$, let $V_{i},W_{i}\subseteq\mathbb{R}^{j+1}$ be $j$-dimensional subspaces that satisfy

[TABLE]

where, for notational simplicity, we write vertical concatenation in a row. Last, for each $\ell\in[k]$, define a center

[TABLE]

Lemma 9.2**.**

When $\|y\|_{2}=1$ and $L^{2}\geq\max_{i}\|A^{(\ell)}_{i}\|_{2}$, it holds that $\operatorname{cost}(X,\mathcal{C}_{\ell})=\Phi(A^{(\ell)}y/\|w_{\ell}\|_{2})$.

Proof.

One can readily verify, using Lemma 9.1, that $P_{ij}\perp v_{i}$ whenever $i\neq\ell$, and thus $P_{ij}\in V_{i}^{\prime}$ and $\operatorname{dist}(P_{ij},X_{\ell})=0$ for $i\neq\ell$.

On the other hand, for $i\neq\ell$,

[TABLE]

and

[TABLE]

Hence when $L^{2}\geq\|y\|_{2}\max_{i}\|A^{(\ell)}_{i}\|_{2}$, it must hold that $W_{\ell}$ is the subspace in $X_{\ell}$ that is the closest to $P_{\ell j}$ for all $j$, and therefore

[TABLE]

Theorem 9.3**.**

Suppose that there exists a function $\Phi$ and absolute constants $C_{0}$ and $\varepsilon_{0}$ such that for any $d\geq C_{0}\log(k/\varepsilon)$ and $\varepsilon\in(0,\varepsilon_{0})$, solving the subspace sketch problem for $\Phi$ requires $M$ bits. Then there exists an absolute constant $C_{1}$ such that for any $k\geq 1$ and $j\geq C_{1}\log(k/\varepsilon)$, any coreset for projective clustering for $\Phi$ requires $kM$ bits.

Proof.

We prove this theorem by a reduction from the subspace sketch problem for $\Phi$ to coresets for projective clustering for $\Phi$.

Choose $D=O(\log k)$ and $d^{\prime}:=j+1-D=C_{0}\log(1/\varepsilon)$. Let $A^{(1)},\dots,A^{(k)}\in\mathbb{R}^{n\times d^{\prime}}$ be $k$ independent hard instances for the subspace sketch problem for $\Phi$. Let $X$ be as constructed before Lemma 9.2. If one can compute a projective clustering coreset for $X$ so that one can approximate $\operatorname{cost}(X,\mathcal{C}_{\ell})$ up to a $(1\pm\varepsilon)$-factor, it follows from Lemma 9.2 that one can approximate $\Phi(A^{(\ell)}y/\|w\|_{2})$ up to a $(1\pm\varepsilon)$-factor for every $\ell\in[k]$ and every unit vector $y\in\mathbb{R}^{d^{\prime}}$. Solving the subspace sketch problem for $\Phi$ for each $A^{(\ell)}$ requires $M$ bits. Therefore, solving $k$ independent instances requires $kM$ bits. ∎

We have the following immediate corollary.

Corollary 9.4**.**

Under the assumptions of Theorem 9.3, any coreset for projective clustering requires $\Omega(jM/\log(k/\varepsilon))$ bits.

Proof.

Let $b=j/(C_{0}\log(k/\varepsilon))$. Let $X^{\prime}$ be a block diagonal matrix of $b$ blocks, each diagonal block is an independent copy of the hard instance $X$ in Theorem 9.3. It then follows from Theorem 9.3 that the lower bound is $\Omega(bM)$ bits. ∎

A lower bound of $\Omega(jk/(\varepsilon^{2}\log k\cdot\operatorname{polylog}(1/\varepsilon))$ follows immediately for $\Phi(x)=\|x\|_{p}^{p}$ (Theorem 3.12) for $p\in[0,+\infty)\setminus 2\mathbb{Z}^{+}$, and the $M$-estimators in Corollary 8.2 and Corollary 8.4.

10 Upper Bounds for the Tukey Loss $p$-Norm

We shall prove in this section an $\widetilde{O}(1/\varepsilon^{2})$ upper bound for estimating a mollified version of the Tukey loss $1$-norm $\Phi(x)$ for a vector $x\in\mathbb{R}^{n}$.

10.1 Mollification of Tukey Loss Function

Consider the classic “bump” function $\psi$ and the standard scaled version $\psi_{t}$, which are defined as

[TABLE]

where $C_{\psi}$ is the normalization constant such that $\int_{-1}^{1}\psi(x)dx=1$. The following is a result on the decay of its Fourier transform. We define the Fourier transform of a function $f\in L^{1}(\mathbb{R})$ to be $\hat{f}(t)=\int_{\mathbb{R}}e^{itx}f(x)dx$, and thus $f(x)=(2\pi)^{-1}\int_{\mathbb{R}}e^{-itx}\hat{f}(t)dt$ if $\hat{f}\in L^{1}(\mathbb{R})$.

Lemma 10.1** ([18]).**

There exists an absolute constant $C>0$ such that $|\widehat{\psi}(t)|\leq C|t|^{-3/4}e^{-\sqrt{|t|}}$.

This enables us to upper bound the derivatives of $\psi$. This is probably a classical result but we do not know an appropriate reference and so we reproduce the proof here.

Lemma 10.2**.**

There exist absolute constants $C_{1},C_{2}>0$ such that $\left\|\psi^{(k)}\right\|_{\infty}\leq C_{1}(C_{2}k\log k)^{2k+2}$.

Proof.

First note that

[TABLE]

On the one hand, taking $T=C_{1}(k\log k)^{2}$, it follows from Lemma 10.1 that

[TABLE]

where $C_{1},C_{2}>0$ are absolute constants. On the other hand, $|\hat{\psi}(t)|\leq\|\psi\|_{1}=1$ for all $t$. Therefore

[TABLE]

The result then follows. ∎

Recall that the Tukey $1$-loss function is

[TABLE]

It is easy to verify that $(\phi\ast\psi_{\tau/4})(x)=\phi(x)$ when $\tau/4\leq|x|\leq 3\tau/4$, and thus if we define

[TABLE]

it would be clear that $\widetilde{\phi}$ is infinitely times differentiable on $(0,\infty)$. Also observe that $\widetilde{\phi}(x)=\phi(x)=\tau$ when $|x|\geq 5\tau/4$, we see that $\widetilde{\phi}$ just mollifies $\phi$ on $[3\tau/4,5\tau/4]$. We shall take $\widetilde{\phi}$ to be our mollified version of the Tukey $1$-loss function.

Next we bound the derivatives of $\widetilde{\phi}$.

Lemma 10.3**.**

There exist absolute constants $C_{1},C_{2}>0$ such that $\left|\widetilde{\phi}^{(k)}(x)\right|\leq C_{1}\tau(C_{2}k^{2}\log^{2}k/\tau)^{k+1}$ for $x\in[3\tau/4,5\tau/4]$.

Proof.

Observe that

[TABLE]

The result follows from Lemma 10.2. ∎

As a corollary of the preceding proposition, we have

Lemma 10.4**.**

Let $a\in(0,\frac{3}{4})$ and $b>1$ be constants. There exists a polynomial $p(x)$ of degree $O(\frac{b-a}{\tau}\log^{2}\frac{1}{\varepsilon a\tau}\log^{2}\log\frac{1}{\varepsilon a\tau})$ such that $|p(x)-\widetilde{\phi}(x)|\leq\varepsilon\widetilde{\phi}(x)$ on $[a\tau,b\tau]$.

Proof.

Since $\widetilde{\phi}(x)\geq a\tau$ on $[a\tau,b\tau]$, it is sufficient to consider the uniform approximation $|p(x)-\widetilde{\phi}(x)|\leq\varepsilon(a\tau)$ on $[a\tau,b\tau]$. It follows from Lemma 2.8 that when $n>k$,

[TABLE]

Invoking Lemma 10.3, we obtain for $n\geq 2k$ that

[TABLE]

where $C_{1},C_{2}>0$ are absolute constants. It is now clear that we can take $k=O(\log\frac{1}{\varepsilon a\tau})$ and $n=O(\frac{b-a}{\tau}k^{2}\log^{2}k)$ so that $E_{n}(\widetilde{\phi};[a\tau,b\tau])\leq\varepsilon\cdot a\tau$. ∎

10.2 Estimation Algorithm

Since $\widetilde{\phi}(x)$ agrees with $|x|$ for small $|x|$, it follows from Theorem 8.1 that solving the subspace sketch problem for $\widetilde{\Phi}(x)$ requires $\widetilde{\Omega}(d/\varepsilon^{2})$ bits. In this subsection we show that this lower bound is tight up to polylogarithmic factors. Specifically we have the following theorem.

Theorem 10.5**.**

Let $\widetilde{\Phi}(x)$ be the mollified Tukey loss $1$-norm of $x\in\mathbb{R}^{n}$. There exists a randomized algorithm which returns an estimate $Z$ to $\widetilde{\Phi}(x)$ such that $(1-\varepsilon)\widetilde{\Phi}(x)\leq Z\leq(1+\varepsilon)\widetilde{\Phi}(x)$ with probability at least $0.9$. The algorithm uses $\widetilde{O}(1/\varepsilon^{2})$ bits of space.

This theorem implies an $\widetilde{O}(d/\varepsilon^{2})$ upper bound for the corresponding subspace sketch problem. The remaining of the section is devoted to the proof of this theorem.

We shall first sample rows of $A$ with sampling rate $\Theta\left(\frac{\tau}{\widetilde{\Phi}(x)\varepsilon^{2}}\right)$. However, we do not know $\widetilde{\Phi}(x)$ in advance. To implement this, we sample rows of $A$ using $O(\log n)$ different sampling rates $1,(1.1)^{-1},(1.1)^{-2},\ldots,1.1^{-O(\log n)}$, and in parallel, estimate $\widetilde{\Phi}(x)$ using a separate data structure of $O(\operatorname{polylog}(n)\cdot d)$ space [9, 8], which gives an estimate $F$ satisfying $0.9\widetilde{\Phi}(x)\leq F\leq 1.1\widetilde{\Phi}(x)$. Then we choose a sampling rate $r=1.1^{-s}$ for some integer $s$ that is closest to $\frac{\tau}{F\varepsilon^{2}}$. Thus $r\in\left[\frac{\tau}{2\Phi(x)\varepsilon^{2}},\frac{2\tau}{\widetilde{\Phi}(x)\varepsilon^{2}}\right]$ when $\widetilde{\Phi}(x)>\frac{\tau}{2\varepsilon^{2}}$, and $r=1$ otherwise.

Now we show that for the chosen sampling rate $r$, the sampled entries give an accurate estimation to $\widetilde{\Phi}(x)$. This is definitely true when $r=1$, in which case there is no sampling at all. Otherwise, let $X_{i}=\widetilde{\Phi}(x_{i})$ if item $i$ is sampled and $X_{i}=0$ otherwise. Let $X=\sum_{i}X_{i}$ and $Z=(1/r)X$. It is clear that $\operatorname{\mathbb{E}}[Z]=\widetilde{\Phi}(x)$. We calculate the variance below.

[TABLE]

It follows from Chebyshev’s inequality that with constant probability,

[TABLE]

We condition on this event in the rest of the proof. Thus, it suffices to estimate the summation of $\widetilde{\Phi}(x_{i})$ for those $x_{i}$ that are sampled. In the rest of this section, we use $L$ to denote the indices of entries that are sampled at the sampling rate $r$.

For each $i\in L$ with $|x_{i}|\geq\tau$, we claim that

[TABLE]

where $x_{L}$ denotes the vector $x$ restricted to the indices in $L$ and $v_{-k}$ denotes the vector $v$ after zeroing out the largest $k$ entries in magnitude.

We first show that $\widetilde{\Phi}(x_{L})=O\left(\frac{\tau}{\varepsilon^{2}}\right)$, which is clearly true when $r=1$, since in this case, $\widetilde{\Phi}(x_{L})=\widetilde{\Phi}(x)=O\left(\frac{\tau}{\varepsilon^{2}}\right)$. When $r<1$, $\sum_{i\in L}\widetilde{\Phi}(x_{i})=(1\pm O(\varepsilon))\cdot r\cdot\widetilde{\Phi}(x)=O\left(\frac{\tau}{\varepsilon^{2}}\right)$.

Let $L^{\prime}=\{i\in L:|x_{i}|\geq\tau/2\}$. It follows that $|L^{\prime}|\leq\widetilde{\Phi}(x_{L})/(\tau/2)=O(1/\varepsilon^{2})$. Hence

[TABLE]

establishing (19).

Therefore, to find all $i\in L$ with $|x_{i}|\geq\tau$, we use an $\ell_{p}$-heavy hitter data structure, which can be realized by a Count-Sketch structure [11] which hashes $x_{L}$ into $O(1/\beta)$ buckets and finds $\beta$-heavy hitters relative to $\|(x_{L})_{-1/\beta}\|_{1}$. Set $\beta=\Theta(\varepsilon^{2})$. In the end we obtain a list $H\subseteq L$ such that every $i\in H$ is a $\beta/2$-heavy hitter relative to $\|(x_{L})_{-1/\beta}\|_{1}$, and all $\beta$-heavy hitters are in $H$. Furthermore, for each $i\in H$ the data structure also returns an estimate $\hat{x}_{i}$ such that $|x_{i}|/2\leq|\hat{x}_{i}|\leq 2|x_{i}|$ whenever $|x_{i}|\geq\tau/2$. The data structure has space complexity $\widetilde{O}(1/\varepsilon^{2})$.

For each $x_{i}\in L$ with $|x_{i}|\geq\tau$, it must hold that $i\in H$. Let $H_{1}=\{i\in H:|\hat{x}_{i}|\geq 5\tau/4\}$ and $H_{2}=\{i\in H:3\tau/8\leq|\hat{x}_{i}|\leq 5\tau/2\}$.

For each $i\in H_{1}$, by the estimation guarantee it must hold that $|x_{i}|\geq 5\tau/4$. Hence $S_{1}=\tau|H_{1}|=\widetilde{\Phi}(x_{H_{1}})$.

For each $i\in H_{2}$ it must hold that $|x_{i}|\in[\frac{3}{16}\tau,5\tau]$, and thus

[TABLE]

Let $p(x)$ be a polynomial such that $|p(x)-\widetilde{\phi}(x)|\leq\varepsilon\widetilde{\phi}(x)$ on $[\frac{3}{16}\tau,5\tau]$. By Lemma 10.4, it is possible to achieve $\deg p=O(\log^{3}(1/(\varepsilon\tau)))$. We now use an estimation algorithm analogous to the HighEnd structure in [21], which uses the same space $\widetilde{O}(1/\varepsilon^{2})$. Using the same BasicHighEnd structure in [21], with constant probability, for each $x_{i}\in H$ we have $T$ estimates $\hat{x}_{i,1},\dots,\hat{x}_{i,T}\in\mathbb{C}$ such that $\hat{x}_{i,t}=x_{i}+\delta_{i,t}$, where each $\delta_{i,t}\in\mathbb{C}$ satisfies $|\delta_{i,t}|\leq|x_{i}|/2$ and $\operatorname{\mathbb{E}}(\delta_{i,t})^{k}=0$ for $k=1,\dots,3\deg p$. The estimator is

[TABLE]

It follows from the analysis in [21] that ($x$ can be replaced with $x_{[n]\setminus H_{1}}$ in the analysis of the variance) that the algorithm will output, with a constant probability,

[TABLE]

For each $i\in[n]\setminus(H_{1}\cup H_{2})$, it must hold that $|x_{i}|\leq\tau/2$ and thus we can use an $\ell_{1}$ sketch algorithm as in [21], and obtain $S_{3}=(1\pm\varepsilon)\widetilde{\Phi}(x_{[n]\setminus(H_{1}\cup H_{2})})$.

Finally, the algorithm returns $S_{1}+S_{2}+S_{3}$, which is a $(1\pm 10\varepsilon)$-approximation to $\widetilde{\Phi}(x)$. Rescaling $\varepsilon$ proves the correctness of the estimate.

For the part of evaluating $S_{2}$ and $S_{3}$, the space complexity is the same as the HighEnd and $\ell_{p}$ sketch algorithm in [21], which are both $\widetilde{O}(1/\varepsilon^{2})$ bits.

11 An Upper Bound for $\ell_{1}$ Subspace Sketches in Two Dimensions

In this section, we prove an $O(\operatorname{polylog}(n)/\varepsilon)$ upper bound for the $\ell_{1}$ subspace sketch problem when $d=2$. Our plan is to reduce the $\ell_{1}$ subspace sketch problem with $d=2$ to coresets for the weighted $1$-median problem with $d=1$. For the latter problem, an $O(\operatorname{polylog}(n)/\varepsilon)$ upper bound is known [16].

For the special case where the first column of the $A$ matrix is all ones, the $\ell_{1}$ subspace sketch problem with $d=2$ is equivalent to coresets for $1$-median with $d=1$. To see this, by homogeneity, we may assume $x_{2}=1$ for the query vector $x\in\mathbb{R}^{2}$. Thus, $\|Ax\|_{1}=\sum_{i=1}^{n}|x_{1}+A_{i,2}|$, which is the $1$-median cost of using $x_{2}$ as the center on $\{-A_{1,2},-A_{2,2},\ldots,-A_{n,2}\}$. When entries of the first column of $A$ are positive but not necessarily all ones, we have

[TABLE]

which is the weighted $1$-median cost of using $x_{1}$ as the center on $\{-A_{1,2}/A_{1,1},-A_{2,2}/A_{2,1},\ldots,-A_{n,2}/A_{n,1}\}$, with weights $\{A_{i,1},A_{i,2},\ldots,A_{n,2}\}$. It has been shown in [16, Theorem 2.8] that there exists a coreset of size $O(\operatorname{polylog}(n)/\varepsilon)$ for the weighted $1$-median problem when $d=1$.

For general $A$, we divide the rows of $A$ into three separate matrices $A^{+}$, $A^{-}$ and $A^{0}$. Here, all entries in the first column of $A^{+}$ are positive, all entries in the first column of $A^{-}$ are negative, and all entries in the first column of $A^{0}$ are zeroes. Since $\|Ax\|_{1}=\|A^{+}x\|_{1}+\|A^{-}x\|_{1}+\|A^{0}x\|_{1}$, we can design subspace sketches separately for $A^{+}$, $A^{-}$ and $A^{0}$. Our reduction above implies an $O(\operatorname{polylog}(n)/\varepsilon)$ upper bound for $A^{+}$ and $A^{-}$. For $A^{0}$, since all entries in the first column are all zero, we have

[TABLE]

Thus, it suffices to store $\sum_{i}|A^{0}_{i,2}|$ for $A^{0}$.

Theorem 11.1**.**

The $\ell_{1}$ subspace sketch problem can be solved using $\widetilde{O}(1/\varepsilon)$ bits when $d=2$.