Near neighbor preserving dimension reduction for doubling subsets of $\ell_1$

TL;DR

This paper introduces a new dimension reduction technique for doubling subsets of  that preserves near neighbor relationships, addressing a gap in metric embedding theory and improving high-dimensional data analysis.

Contribution

It presents the first near neighbor-preserving embedding for doubling subsets of , using covering sets and random projections with Cauchy variables.

Findings

Provides a dimension reduction method with bounded distortion for doubling  subsets.

Analyzes tradeoffs between different covering set constructions.

Employs concentration bounds for Cauchy variables that are of independent interest.

Abstract

Randomized dimensionality reduction has been recognized as one of the fundamental techniques in handling high-dimensional data. Starting with the celebrated Johnson-Lindenstrauss Lemma, such reductions have been studied in depth for the Euclidean $(\ell_2)$ metric, but much less for the Manhattan $(\ell_1)$ metric. Our primary motivation is the approximate nearest neighbor problem in $\ell_1$. We exploit its reduction to the decision-with-witness version, called approximate \textit{near} neighbor, which incurs a roughly logarithmic overhead. In 2007, Indyk and Naor, in the context of approximate nearest neighbors, introduced the notion of nearest neighbor-preserving embeddings. These are randomized embeddings between two metric spaces with guaranteed bounded distortion only for the distances between a query point and a point set. Such embeddings are known to exist for both $\ell_2$ and $\ell_1$ metrics, as well as for doubling subsets of $\ell_2$. The case that remained open were doubling subsets of $\ell_1$. In this paper, we propose a dimension reduction by means of a \textit{near} neighbor-preserving embedding for doubling subsets of $\ell_1$. Our approach is to represent the pointset with a carefully chosen covering set, then randomly project the latter. We study two types of covering sets: $c$-approximate $r$-nets and randomly shifted grids, and we discuss the tradeoff between them in terms of preprocessing time and target dimension. We employ Cauchy variables: certain concentration bounds derived should be of independent interest.

Equations

$$k={\left(\log {\lambda }_P\cdot \log (c/\epsilon )\right)}^{\Theta (1/\epsilon )}/\zeta (\epsilon ),$$

$$k={\left(\log {\lambda }_P\cdot \log (d/\epsilon )\right)}^{\Theta (1/\epsilon )}/\zeta (\epsilon ),$$

$$\Vert f(p)-f(q){\Vert }_1\in (2/\delta )\cdot \Vert p-q{\Vert }_1\pm 1.$$

$$\Vert p-q{\Vert }_1\le 1$$

$$\Vert p-q{\Vert }_1\ge c$$

$$\le \delta ,$$

$$\le \frac{1+\gamma }{1+\epsilon }.$$

$$E\left[|X{|}^{1/2}\right]=\frac{2}{\pi }{\int }_0^{\infty }\frac{\sqrt{x}}{1+x^2}\text{leavevmode}\mathrm{d}x=\frac{2}{\pi }\frac{\pi }{\sqrt{2}}=\sqrt{2}.$$

$$E\left[\exp (-\beta |X{|}^{1/2})\right]\le \frac{2}{\beta }.$$

$${\int }_0^1{\mathrm{e}}^{-\beta x^{1/2}}\text{leavevmode}\mathrm{d}x=\frac{2}{{\beta }^2}\left(1-\frac{\beta +1}{{\mathrm{e}}^{\beta }}\right).$$

$$E\left[\exp (-\beta |X{|}^{1/2})\right]$$

$$=\frac{2}{\pi }{\int }_0^1{\mathrm{e}}^{-\beta x^{1/2}}\cdot \frac{1}{1+x^2}\text{leavevmode}\mathrm{d}x+\frac{2}{\pi }{\int }_1^{\infty }{\mathrm{e}}^{-\beta x^{1/2}}\cdot \frac{1}{1+x^2}\text{leavevmode}\mathrm{d}x$$

$$\le \frac{2}{\pi }{\int }_0^1{\mathrm{e}}^{-\beta x^{1/2}}\text{leavevmode}\mathrm{d}x+\frac{2}{\pi }{\int }_1^{\infty }{\mathrm{e}}^{-\beta }\cdot \frac{1}{1+x^2}\text{leavevmode}\mathrm{d}x$$

$$=\frac{2}{\pi }\cdot \frac{2}{{\beta }^2}\left(1-\frac{\beta +1}{{\mathrm{e}}^{\beta }}\right)+\frac{1}{2{\mathrm{e}}^{\beta }}$$

$$\le \frac{4}{\pi {\beta }^2}+\frac{1}{2{\mathrm{e}}^{\beta }}$$

$$\le \frac{2}{\beta }.$$

$${\Vert x\Vert }_p\le {\Vert x\Vert }_q\le d^{1/q-1/p}{\Vert x\Vert }_p.$$

$$=$$

$$\le \frac{E[\exp (-\beta \tilde{S})]}{\exp (-\beta E[\tilde{S}]/D)}$$

$$=\frac{E[\exp (-\beta |X_j{|}^{1/2}){]}^k}{\exp (-\beta \sqrt{2}k/D)}$$

$$\le {\left(\frac{2}{\beta }\right)}^k\cdot {\mathrm{e}}^{\sqrt{2}\beta k/D}.$$

$$\forall s\in \mathcal{N}:\text{leavevmode}{\Vert s-q\Vert }_1\ge D_0⟹{\Vert f(s)-f(q))\Vert }_1\ge 4.$$

$$N_i:=\{s\in \mathcal{N}|D_i\le {\Vert s\Vert }_1<D_{i+1}\},D_i=2^{2i}{D}_0,i=0,1,2,\dots$$

$$|N_i|\le {\lambda }_P^{\lceil \log \left(4c{D}_{i+1}/\epsilon \right)\rceil }\le {\lambda }_P^{4\log \left(c{D}_{i+1}/\epsilon \right)}.$$

$$=$$

$$\le \sum _{i=0}^{\infty }|N_i|$$

$$=\sum _{i=0}^{\infty }|N_i|$$

$$\sum _{i=0}^{\infty }|N_i|$$

$$=\sum _{i=0}^{\infty }\frac{2^{\log ({\lambda }_P)(4\log (c{D}_0/\epsilon )+2i+2)}}{2^{k(i+1)}}$$

$$\le \sum _{i=0}^{\infty }\frac{2^{\log ({\lambda }_P)\cdot 4\log (c{D}_0/\epsilon )}\cdot 2^{2\log ({\lambda }_P)(i+1)}}{2^{(4\cdot \log {\lambda }_P\cdot \log (c{D}_0/\epsilon ))(i+1)}\cdot 2^{2\log (2{\lambda }_P/\delta ))(i+1)}}$$

$$\le \sum _{i=0}^{\infty }{2}^{-2\log (2/\delta ))(i+1)}$$

$$=\sum _{i=0}^{\infty }{\left(\frac{{\delta }^2}{4}\right)}^i-1$$

$$=\frac{{\delta }^2}{4-{\delta }^2}$$

$$\le \delta .$$

$$k>C\left(\log {\lambda }_P\cdot \log (c\log k/\epsilon )+\log (1/\delta )\right)>4\cdot \log {\lambda }_P\cdot \log (c{D}_0/\epsilon )+2\log (2{\lambda }_P/\delta )$$

$${\Vert h(p^{\ast })-f(q)\Vert }_1\le 1+3\epsilon ,$$

$$\forall p\in P:\text{leavevmode}\Vert p-q{\Vert }_1>1+9\epsilon ⟹{\Vert h(p)-f(q)\Vert }_1>1+3\epsilon .$$

$$\forall p\in P:\text{leavevmode}{\Vert p-q\Vert }_1\ge D_0+\epsilon ⟹{\Vert h(p)-f(q)\Vert }_1\ge 4.$$

$$\forall p\in P:\text{leavevmode}\Vert p-q{\Vert }_1\in (1+9\epsilon ,D_0+\epsilon )⟹{\Vert h(p)-f(q)\Vert }_1>(1+8\epsilon )(1-\epsilon )\ge 1+3\epsilon .$$

$$\displaystyle k\geq\frac{\big{(}\ln{(5\lambda_{P}^{8\log{(cD_{0}/\varepsilon)}}/\varepsilon)}\big{)}^{2/\varepsilon}}{\zeta(\varepsilon)}=\frac{\big{(}\Theta(\log\log k\cdot\log{\lambda_{P}}+\log{\lambda_{P}}\cdot\ln(c/\varepsilon))\big{)}^{2/\varepsilon}}{\zeta(\varepsilon)}.$$

Full text

Near-Neighbor Preserving Dimension Reduction for Doubling Subsets of $\ell_{1}$

Ioannis Z. Emiris

  

Vasilis Margonis

  

Ioannis Psarros111This work was done while the third author was a PhD candidate in the Department of Informatics & Telecommunications, National & Kapodistrian University of Athens, Greece

1 Introduction

Proximity search is a fundamental computational problem with several applications in Computer Science and beyond. Proximity problems in metric spaces of low dimension have been typically handled by methods which discretize the space and therefore are affected by the curse of dimensionality, making them unfit for high-dimensional spaces. In the past two decades, the increasing need for analyzing high-dimensional data led researchers to devise randomized and approximation algorithms with polynomial dependence on the dimension.

A fundamental proximity problem is Approximate Nearest Neighbor search. By known reductions [11], one can (up to polylogarithmic factors) focus on the decision version with witness, namely the $(c,R)$-Approximate Near Neighbor problem:

Definition 1.1** (Approximate Near Neighbor).**

Let $(X,d_{X})$ be a metric space. Given $P\subseteq X$ and reals $R>0$, $c\geq 1$, build a data structure $\mathcal{S}$ that, given a query point $q\in X$, performs as follows:

• •

If the nearest neighbor of $q$ lies within distance at most $R$, then $\mathcal{S}$ is allowed to report any point $p^{*}\in P$ such that $d_{X}(q,p^{*})\leq cR$.

• •

If all points lie at distance more than $cR$ from $q$, then $\mathcal{S}$ should return $\bot$.

In general, $\mathcal{S}$ returns either a point at distance $\leq cR$ or $\bot$, even when none of the above two cases occurs.

From now on, we assume $R=1$ because we can re-scale the data set, and we refer to this problem as $c$-ANN, or simply ANN. We focus on subsets of $\ell_{1}^{d}$: the input dataset consists of $n$ vectors in $\mathbb{R}^{d}$ and the distance function is the standard $\ell_{1}$ norm $\|\cdot\|_{1}$. Note that all logarithms are base $2$.

Previous work.

Some highlights in the study of data structures for high-dimensional normed spaces are the various variants, proofs, and applications of the Johnson Lindenstrauss Lemma (e.g. [1, 2, 3]), sketches based on $p$-stable distributions [14], and Locality Sensitive Hashing (e.g. [15, 4, 5]). In the core of most high-dimensional solutions lies the fact that for certain metric spaces e.g. $\ell_{p},p\in[1,2]$, the distance can be efficiently sketched. Spaces which are considered to be harder in this context, such as $\ell_{\infty}$, can also be treated [13], and are very interesting since they can be used as host spaces for various norms [6].

Significant amount of work has been undertaken for pointsets of low doubling dimension, since it is today one of the primary paradigms for capturing input structure (formal definitions in the next section). For any finite metric space $X$ of doubling dimension $\mathrm{dim}(X)$, there exists a data structure [12, 9] with expected preprocessing time $O(2^{\mathrm{dim}(X)}n\log{n})$, space usage $O(2^{\mathrm{dim}(X)}n)$ (or even $O(n)$) and query time $O(2^{\mathrm{dim}(X)}\log{n}+\varepsilon^{-O(\mathrm{dim}(X)})$.

In [16], they introduced the notion of nearest-neighbor preserving embeddings, and it was proven that in this context one can achieve dimension reduction for doubling subsets of $\ell_{2}$, with the target dimension depending only on the dataset’s doubling dimension. Even before, Indyk [14] had introduced a randomized embedding for dimension reduction in $\ell_{1}$, which is suitable for proximity search purposes, and it achieves target dimension polylogarithmic in the size of the pointset. Naturally, such approaches can be easily combined with any known data structure to be used in the projection space. Randomized embeddings have been recently used in the ANN context [8], for doubling subsets of $\ell_{p}$, $2<p<\infty$.

It is known that dimension reduction in $\ell_{1}$ cannot be achieved in the same generality as in $\ell_{2}$, even assuming that the pointset is of low doubling dimension [LMN05]: there are arbitrarily large $n$-point subsets $P\subseteq\ell_{1}$ which are doubling with constant $6$, such that every embedding with distortion $D$ of $P$ into $\ell_{1}^{k}$ requires dimension $n^{\Omega(1/D^{2})}$. Aiming for more restrictive guarantees, e.g. preserving distances within some pre-defined range, is a relevant workaround. Then, dimension reduction techniques for doubling subsets of $\ell_{p}$, $p\in[1,2]$, exist [7], but they rely on partition algorithms which require the whole pointset to be known in advance. Hence, applicability of such techniques is quite limited and, specifically, it is not clear whether they can be used in an online setting where query points are not known beforehand.

Contribution.

In this paper, we establish two non-linear near neighbor-preserving embeddings for doubling subsets of $\ell_{1}^{d}$. We use a definition which is essentially a modified version of the nearest neighbor preserving embedding of [16]: the guarantees which are required are weaker since we consider the decision version of the problem, therefore the embedding depends on some range parameter $R>0$.

Definition 1.2** (Near-neighbor preserving embedding).**

Let $(Y,d_{Y})$, $(Z,d_{Z})$ be metric spaces and $X\subseteq Y$. A distribution over mappings $f:Y\rightarrow Z$ is a near-neighbor preserving embedding with range $R>0$, distortion $D\geq 1$ and probability of correctness $\mathcal{P}\in[0,1]$ if for every $\alpha\geq D$ and any $q\in Y$, if $x\in X$ is such that $d_{Y}(x,q)\leq R$, then with probability at least $\mathcal{P}$,

• •

$d_{Z}(f(x),f(q))\leq D\cdot R$,

• •

$\forall p\in X:\leavevmode\nobreak\ d_{Y}(p,q)>D\cdot\alpha\cdot R\implies d_{Z}(f(p),f(q))>\alpha\cdot R$.

Considering a pointset $P\subset\ell_{1}^{d}$ of cardinality $n$, our results concern $\ell_{1}^{k}$ as the target space, where $k$ depends on the doubling dimension of $P$. We assume that $R=1$, since we can rescale the dataset. More specifically:

In Theorem 4.3, we prove that for every $\varepsilon\in(0,1/2)$ and $c\geq 1$, there is a randomized mapping $h:\ell_{1}^{d}\rightarrow\ell_{1}^{k}$ that can be computed in time $\tilde{O}(dn^{1+1/\Omega(c)})$ and is near neighbor-preserving for $P$ with distortion $1{+}6\varepsilon$ and probability of correctness $\Omega(\varepsilon)$, where

[TABLE]

for a function $\zeta(\varepsilon)>0$ depending only on $\varepsilon$. Although the mapping $h$ depends on the pointset, the parameter $c$ is user-defined and therefore provides a trade-off between preprocessing time and target dimension. 2. 2.

In Theorem 5.5, we show that for every $\varepsilon\in(0,1/2)$, there is a randomized mapping $h^{\prime}:\ell_{1}^{d}\rightarrow\ell_{1}^{k}$ that can be computed in time $O(dkn)$ and is near neighbor-preserving for $P$ with distortion $1{+}6\varepsilon$ and probability of correctness $\Omega(\varepsilon)$, where

[TABLE]

for a function $\zeta(\varepsilon)>0$ depending only on $\varepsilon$. In this case, the function $h^{\prime}$ is oblivious to $P$ and well-defined over the whole space, but the target dimension depends on $d$.

On the low-preprocessing-time extreme, one can embed the dataset in near-linear time, but the target dimension is polynomial in $\log\log n$. This is to be juxtaposed to the analogous result by Indyk [14], which provides with target dimension polynomial in $\log n$, without any assumption on the doubling dimension of the dataset. On the other hand, one can obtain a preprocessing time of $dn^{1+\delta}$ for any constant $\delta>0$, and target dimension which depends solely on the doubling dimension.

Techniques.

Both embeddings consist of two basic components. First, we represent the pointset $P$ with an $\varepsilon$-covering set, and then we apply a random linear projection à la Indyk [14] to that set, using Cauchy variables.

The role of the covering set is to exploit the doubling dimension of $P$. In the analogous result for $\ell_{2}$ [16], no representative sets were used; the mapping was just a random linear projection of $P$. In the case of $\ell_{1}$ however, a similar analysis of a linear projection with Cauchy variables without these representative sets seems to be impossible, since the Cauchy distribution is heavy tailed.

In Theorem 4.3, we consider $c$-approximate $r$-nets as a covering set. Inspired by the algorithm of [10] for $\ell_{2}$, we design an algorithm that computes a $c$-approximate $r$-net in $\ell_{1}$ in subquadratic –but superlinear– time. On the other hand, Theorem 5.5 relies on randomly shifted grids, which can be computed in linear time, but are inferior to nets in terms of capturing the doubling dimension of the pointset.

To bound the distortion incurred by the randomized projection, we exploit the 1-stability property of the Cauchy distribution. To this end, we prove a concentration bound for sums of independent Cauchy variables that should be of interest beyond the scope of this paper. To overcome the technical difficulties associated with the heavy tails of the Cauchy distribution, we study sums of square roots of Cauchy variables, where in [14], Indyk considers sums of truncated Cauchy variables instead. Although our concentration bound is rather weak, it is sufficient for our purposes and its analysis is much simpler compared to Indyk’s.

Algorithmic implications.

Our results show that efficient dimension reduction for doubling subsets of $\ell_{1}$ is possible, in the context of ANN. In particular, these results imply efficient sketches, meaning that one can solve ANN with minimal storage per point. Dimension reduction also serves as a problem reduction from a high-dimensional hard instance to a low-dimensional easy instance. Since the algorithms presented in this paper are quite simple, they should also be of practical interest: they easily extend the scope of any implementation which has been optimized to solve the problem in low dimension, so that it may handle high-dimensional data.

Our embedding can be combined with the bucketing method of [11] for the $(1{+}\varepsilon)$-ANN problem in $\ell_{1}^{d}$. For instance, setting $c=\log n$ in Theorem 4.3, yields preprocessing time $dn^{1+o(1)}$, space $n^{1+o(1)}$ and query time $O(d){\cdot}(\log{\lambda_{P}}\cdot\log\log n)^{O(1/\varepsilon)}$ assuming that the doubling dimension is a fixed constant. This improves upon existing results: the query time of [KL04] depends on the aspect ratio of the dataset, while the data structures of [12, 9] support queries with time complexity which depends exponentially on the doubling dimension. However, it is worth noting that one could potentially improve the results of [KL04, 12, 9] in the special case of $\ell_{1}$, by employing ANN data structures with fast query time, in order to accelerate the traversal of the net-tree. Hence, while our result gives a simple framework for exploiting the intrinsic dimension of doubling subsets of $\ell_{1}$, it is unlikely that it shall improve upon simple variants of previous results in terms of complexity bounds.

Organization.

The next section introduces basic concepts and some relevant existing results. Section 3 establishes a concentration bound on sums of independent Cauchy variables. Section 4, achieves dimensionality reduction by means of representing the pointset by a carefully chosen net, while Section 5 employs randomly shifted grids for the same task. We conclude with discussion of results and potential improvements.

2 Preliminaries

In this section, we define basic notions about doubling metrics and present useful previous results.

Definition 2.1**.**

Consider any metric space $(X,{\rm d}_{X})$ and let $B(p,r)=\{x\in X\mid{\rm d}_{X}(x,p)\leq r\}$. The doubling constant of $X$, denoted $\lambda_{X}$, is the smallest integer $\lambda_{X}$ such that for any $p\in X$ and $r>0$, the ball $B(p,r)$ can be covered by $\lambda_{X}$ balls of radius $r/2$ centered at points in $X$.

The doubling dimension of $(X,{\rm d}_{X})$ is defined as $\log{\lambda_{X}}$. Nets play an important role in the study of embeddings, as well as in designing efficient data structures for doubling metrics.

Definition 2.2**.**

For $c\geq 1$, $r>0$ and metric space $(V,d_{V})$, a $c$-approximate $r$-net of $V$ is a subset $\mathcal{N}\subseteq V$ such that no two points of $\mathcal{N}$ are within distance $r$ of each other, and every point of $V$ lies within distance at most $c{\cdot}r$ from some point of $\mathcal{N}$.

Theorem 2.3**.**

Let $P\subset\ell_{1}^{d}$ such that $|P|=n$. Then, for any $c>0$, $r>0$, one can compute a $c$-approximate $r$-net of $P$ in time $\tilde{O}(dn^{1+1/c^{\prime}})$, where $c^{\prime}=\Omega(c)$. The result is correct with high probability. The algorithm also returns the assignment of each point of $P$ to the point of the net which covers it.

Proof 2.4**.**

We employ some basic ideas from [11]. An analogous result for $\ell_{2}$ is stated in [10]. First, we assume $r=1$, since we are able to re-scale the point set. Now, we consider a randomly shifted grid with side-length $2$. The probability that two points $p,q\in P$ fall into the same grid cell, is at least $1-\|p-q\|_{1}/2$. For each non-empty grid cell we snap points to a grid: each coordinate is rounded to the nearest multiple of $\delta=1/10dc$. Then, coordinates are multiplied by $1/\delta$ and each point $x=(x_{1},\ldots,x_{d})\in[2\delta]^{d}$ is mapped to $\{0,1\}^{2d/\delta}$ by a function $G$ as follows: $G(x)=(g(x_{1}),\ldots,g(x_{d}))$, where $g(z)$ is a binary string of $z$ ones followed by $2/\delta-z$ zeros. For any two points $p,q$ in the same grid cell, let $f(p)$,$f(q)$ be the two binary strings obtained by the above mapping. Notice that,

[TABLE]

Hence,

[TABLE]

Now, we employ the LSH family of [11], for the Hamming space. After standard concatenation, we can assume that the family is $(\rho,c^{\prime}\rho,n^{-1/c^{\prime}},n^{-1})$-sensitive, where $\rho=\left({2}/{\delta}\right)+1$ and $c^{\prime}=\Omega(c)$. Let $\alpha=n^{-1/c^{\prime}}$ and $\beta=n^{-1}$.

Notice that for the above two-level hashing table we obtain the following guarantees. Any two points $p,q\in P$, such that $\|p-q\|_{1}\leq 1$, fall into the same bucket with probability $\geq\alpha/2$. Any two points $p,q\in P$, such that $\|p-q\|_{1}\geq c$, fall into the same bucket with probability $\leq\beta$.

Finally, we independently build $k=\Theta(n^{1/c^{\prime}}\log n)$ hashtables as above, where the random hash function is defined as a concatenation of the function which maps points to their grid cell id and one LSH function. We pick an arbitrary ordering $p_{1},\ldots,p_{n}\in P$. We follow a greedy strategy in order to compute the approximate net. We start with point $p_{1}$, and we add it to the net. We mark all (unmarked) points which fall at the same bucket with $p_{1}$, in one of the $k$ hashtables, and are at distance $\leq cr$. Then, we proceed with point $p_{2}$. If $p_{2}$ is unmarked, then we repeat the above. Otherwise, we proceed with $p_{3}$. The above iteration stops when all points have been marked. Throughout the procedure, we are able to store one pointer for each point, indicating the center which covered it.

Correctness.

The probability that a good pair $p,q$ does not fall into the same bucket for any of the $k$ hashtables is $\leq\left(1-\alpha/2\right)^{k}\leq n^{-10}$. Hence, with high probability, the packing property holds, and the covering property holds because the above algorithm stops when all points are marked.

Running time.

*The time to build the $k$ hashtables is $k\cdot n=\tilde{O}(n^{1+1/c^{\prime}})$. Then, at most $n$ queries are performed: for each query, we investigate $k$ buckets and the expected number of false positives is $\leq k\cdot n^{2}\cdot\beta=\tilde{O}(n^{1+1/c^{\prime}})$. Hence, if we stop after having seen a sufficient amount of false positives, we obtain time complexity $\tilde{O}(n^{1+1/c^{\prime}})$ and the covering property holds with constant probability. We can repeat the above procedure $O(\log n)$ times to obtain high probability of success. *

The main result in the context of randomized embeddings for dimension reduction in $\ell_{1}^{d}$ is the following theorem, which exploits the $1$-stability property of Cauchy random variables and provides with an asymmetric guarantee: The probability of non-contraction is high, but the probability of non-expansion is constant. Nevertheless, this asymmetric property is sufficient for proximity search.

Theorem 2.5** (Thm 5, [14]).**

For any $\varepsilon\leq 1/2$, $\delta>0$, $\varepsilon>\gamma>0$ there is a probability space over linear mappings $f:\ell_{1}^{d}\rightarrow\ell_{1}^{k}$, where $k=(\ln{(1/\delta)})^{1/(\varepsilon-\gamma)}/\zeta(\gamma)$, for a function $\zeta(\gamma)>0$ depending only on $\gamma$, such that for any pair of points $p,q\in\ell_{1}^{d}$:

[TABLE]

Note that the embedding is defined as $f(u)=Au/T$, where $A$ is a $k{\times}d$ matrix with each element being an i.i.d. Cauchy random variable. In addition, $T$ is a scaling factor defined as the expectation of a sum of truncated Cauchy variables, such that $T=\Theta(k\log{(k/\varepsilon)})$ (see Lemma 5 in [14]).

One key observation here is that given a pointset $P$ in a space of bounded aspect ratio $\Phi$, one can directly employ Theorem 2.5. The number of points can be upper bounded by a function of $\lambda_{P}$ and $\Phi$, and hence the new dimension, $k$, depends only on these parameters. This paper proves better bounds than the ones of Theorem 2.5 for doubling subsets of $\ell_{1}^{d}$, without any assumption on the aspect ratio.

3 Concentration bounds for Cauchy variables

In this section, we prove some basic properties of the Cauchy distribution, which serves as our main embedding tool.

Let $C_{\mathcal{D}}$ denote the Cauchy distribution with density $c(x)=(1/\pi)/(1+x^{2})$. One key property of the Cauchy distribution is the so-called $1$-stability property: Let $v=(v_{1},\ldots,v_{k})\in\mathbb{R}^{k}$ and $X_{1},\ldots,X_{k}$ be i.i.d. random variables following $C_{\mathcal{D}}$, then $\sum_{j=1}^{k}X_{i}v_{i}$ is distributed as $X{\cdot}\|v\|_{1}$, where $X\sim C_{\mathcal{D}}$.

The Cauchy distribution has undefined mean. However, for $0<q<1$, the mean of the $q$-th power of a Cauchy random variable can be defined. More specifically, for some $X\sim C_{\mathcal{D}}$ we have

[TABLE]

The following lemma provides a bound for the moment-generating function of $|X|^{1/2}$.

Lemma 3.1**.**

Let $X\sim C_{\mathcal{D}}$. Then for any $\beta>1$:

[TABLE]

Proof 3.2**.**

For any constant $\beta$,

[TABLE]

Then, for any $\beta>1$,

[TABLE]

Let $S:=\sum_{j=1}^{k}|X_{j}|$ where each $X_{j}$ is an i.i.d. Cauchy variable. To prove concentration bounds for $S$, we study the sum $\tilde{S}:=\sum_{j=1}^{k}|X_{j}|^{1/2}$. By Hölder’s Inequality, for any $x\in\mathbb{R}^{d}$ and $p>q>0$,

[TABLE]

Consequently, for $x=(X_{1},\ldots,X_{k})\in\mathbb{R}^{k}$, $p=1$ and $q=1/2$ we have that $S\leq\tilde{S}^{2}\leq k\cdot S$, hence for any $t>0$,

[TABLE]

We use the bound on the moment-generating function, to prove a Chernoff-type concentration bound for $\tilde{S}$, which by Eq. (1) translates into a concentration bound for $S$.

Lemma 3.3**.**

For every $D>1$,

[TABLE]

Proof 3.4**.**

Since $X_{j}$’s are independent, $\operatorname*{\mathbb{E}}[\tilde{S}]=\sqrt{2}k$. Then, by Lemma 3.1 and Markov’s inequality, for any $\beta>1$, it follows that

[TABLE]

Setting $\beta=D$ completes the proof.

4 Net-based dimension reduction

In this section we describe the dimension reduction mapping for $\ell_{1}$ via $r$-nets. Let $P\subset\ell_{1}^{d}$ be a set of $n$ points with doubling constant $\lambda_{P}$. For some point $x\in\mathbb{R}^{d}$ and $r>0$, we denote by $B_{1}(x,r)$ the $\ell_{1}$-ball of radius $r$ around $x$. The embedding is non-linear and is carried out in two steps.

First, we compute a $c$-approximate $(\varepsilon/c)$-net $\mathcal{N}$ of $P$ with the algorithm of Theorem 2.3. Moreover, the algorithm assigns each point of $P$ to the point of $\mathcal{N}$ which covered it. Let $g:P\rightarrow\mathcal{N}$ be this assignment. In the second step, for every $s\in\mathcal{N}$ and any query point $q\in\ell_{1}^{d}$, we apply the linear map of Theorem 2.5. That is, $f(s)=As/T$, where $A$ is a $k{\times}d$ matrix with each element being an i.i.d. Cauchy random variable. Recall that value $T=\Theta(k\log{(k/\varepsilon)})$. By the $1$-stability property of the Cauchy distribution, $f(s)$ is distributed as $\left\rVert{s}\right\rVert_{1}\cdot(Y_{1},\ldots,Y_{k})$, where each $Y_{j}$ is i.i.d. and $Y_{j}\sim C_{\mathcal{D}}$. Hence, $\left\rVert{f(s)}\right\rVert_{1}=\left\rVert{s}\right\rVert_{1}\cdot S$ where $S:=\sum_{j}|Y_{j}|$.

We define the embedding to be $h=f\circ g$. We apply $h$ to every point in $P$, and $f$ to any query point $q$. It is clear from the properties of the net that $g$ incurs an additive error of $\pm\varepsilon$ on the distance between $q$ and any point in $P$, so it is sufficient to consider the distortion of $f$.

Our analysis consists of studying separately the following disjoint subsets of $\mathcal{N}$: Points that lie at distance at most $D_{0}$ from the query and points that lie at distance at least $D_{0}$, for some $D_{0}>1$ chosen appropriately. For the former set, we directly apply Theorem 2.5, as it has bounded diameter.

The next lemma guarantees the low distortion for points of the latter set, namely those that are sufficiently far from the query. We consider the sum of the square roots of each $|Y_{j}|$, i.e., $\tilde{S}=\sum_{j}|Y_{j}|^{1/2}$, in order to employ the tools of Section 3.

Lemma 4.1**.**

Fix a query point $q\in\ell_{1}^{d}$. For any $\varepsilon\leq 1/2$, $c\geq 1$, $\delta\in(0,1)$, there exists $D_{0}=O(\log(k/\varepsilon))$ such that for $k=\Theta\left(\log^{2}{\lambda_{P}}\cdot\log(c/\varepsilon)+\log(1/\delta)\right)$, with probability at least $1-\delta$,

[TABLE]

Proof 4.2**.**

Assume wlog that the query point is the origin $(0,\dots,0)$. For some $D_{0}>1$, we define the following subsets of $\mathcal{N}$:

[TABLE]

By the definition of doubling constant and the fact that two points of $\mathcal{N}$ lie at distance at least $\varepsilon$,

[TABLE]

Therefore, by the union bound, and Eq. (1):

[TABLE]

By Lemma 3.3, for $D_{0}{=}\lceil 800T/k\rceil{=}\Theta(\log(k/\varepsilon))$ and $k>4{\cdot}\log{\lambda_{P}}{\cdot}\log(cD_{0}/\varepsilon)+2\log(2\lambda_{P}/\delta)$:

[TABLE]

Finally, for some large enough constant $C$, we demand that

[TABLE]

which is satisfied for $k=\Theta\left(\log^{2}{\lambda_{P}}\cdot\log(c/\varepsilon)+\log(1/\delta)\right)$.

Theorem 4.3**.**

Let $P\subset\ell_{1}^{d}$ such that $|P|=n$. For any $\varepsilon\in(0,1/2)$ and $c\geq 1$, there is a non-linear randomized embedding $h=f\circ g:\ell_{1}^{d}\rightarrow\ell_{1}^{k}$, where $k=\left(\log{\lambda_{P}}\cdot\log(c/\varepsilon)\right)^{\Theta(1/\varepsilon)}/\zeta(\varepsilon)$, for a function $\zeta(\varepsilon)>0$ depending only on $\varepsilon$, such that, for any $q\in\ell_{1}^{d}$ , if there exists $p^{*}\in P$ such that $\|p^{*}-q\|_{1}\leq 1$, then, with probability $\Omega(\varepsilon)$:

[TABLE]

Set $P$ can be embedded in time $\tilde{O}(dn^{1+1/\Omega(c)})$, and any query $q\in\ell_{1}^{d}$ can be embedded in time $O(dk)$.

Proof 4.4**.**

Let $f,g$ be the mappings defined in the beginning of the section and $D_{0}=\Theta(\log(k/\varepsilon))$. Assume wlog for simplicity that $q=0^{d}$. Then, by Lemma 4.1 for $k=\Theta\left(\log^{2}{\lambda_{P}}\cdot\log(c/\varepsilon)\right)$, with probability at least $1-\varepsilon/5$, we have:

[TABLE]

By Theorem 2.5, for $\gamma=\varepsilon/10$ and $\delta=\varepsilon/(5\lambda_{P}^{8\log{(cD_{0}/\varepsilon)}})$, with probability at least $1-\varepsilon/5$, we get:

[TABLE]

Moreover,

[TABLE]

Then, the target dimension needs to satisfy the following inequality:

[TABLE]

Hence, for $k=\left(\log{\lambda_{P}}\cdot\log(c/\varepsilon)\right)^{\Theta(1/\varepsilon)}/\zeta(\varepsilon)$, we achieve a total probability of success in $\Omega(\varepsilon)$, which completes the proof.

5 Dimension reduction based on randomly shifted grids

In this section, we explore some properties of randomly shifted grids, and we present a simplified embedding which consists of a first step of snapping points to a grid, and a second step of randomly projecting grid points.

Let $w>0$ and $t$ be chosen uniformly at random from the interval $[0,w]$. The function

[TABLE]

induces a random partition of the real line into segments of length $w$. Hence, the function

[TABLE]

for $t_{1},\ldots,t_{d}$ independent uniform random variables in the interval $[0,w]$, induces a randomly shifted grid in $\mathbb{R}^{d}$. For a set $X\subseteq\mathbb{R}^{d}$, we denote by $g_{w}(X)$, the image of $X$ on the randomly shifted grid points defined by $g_{w}$. For some $x\in\mathbb{R}^{d}$ and $r>0$, the number of grid cells of $g_{w}(\ell_{1}^{d})$ that $B_{1}(x,r)$ intersects per axis is independent, and in expectation is $1{+}2r/w$. Then, the expected total number of grid cells that $B_{1}(x,r)$ intersects is at most $(1{+}2r/w)^{d}$.

Now let $P\subset\ell_{1}^{d}$ be a set of $n$ points with doubling constant $\lambda_{P}$ and $q\in\ell_{1}^{d}$ a query point. For $w=\varepsilon/d$, the $\ell_{1}$-diameter of each cell is $\varepsilon$ and therefore $g_{w}(P)$ is an $\varepsilon$-covering set of $P$.

Lemma 5.1**.**

Let $\mathcal{R}>1$ and $P^{\prime}:=B_{1}(q,\mathcal{R})\cap P$. Then, for $w=\varepsilon/d$

[TABLE]

Proof 5.2**.**

By the doubling constant definition, there exists a set of balls of radius $\varepsilon/d^{2}$ centered at points in $P^{\prime}$, of cardinality at most $\lambda_{P}^{2\log(d\mathcal{R}/\varepsilon)}$ which covers $P^{\prime}$. For each ball of radius $\varepsilon/d^{2}$, the expected number of intersecting grid cells is $(1{+}2/d)^{d}\leq{\rm e}^{2}$. The lemma follows by linearity of expectation.

The next lemma shows that, with constant probability, the growth on the number of representatives, as we move away from $q$, is bounded.

Lemma 5.3**.**

Let $\{D_{i}\}_{i\in\mathbb{N}}$ be a sequence of radii such that, for any $i$, $D_{i+1}=4D_{i}$. Let $A_{i}$ be the points of $g_{w}(P)$ within distance $D_{i+1}=2^{2(i+1)}D_{0}$ from $q$. Then, with probability at least $1/3$,

[TABLE]

Proof 5.4**.**

By Lemma 5.1, $\operatorname*{\mathbb{E}}[|A_{i}|]\leq 8\lambda_{P}^{2\log(dD_{i+1}/\varepsilon)}$ for every $i\in\{-1,0,\ldots\}$. Then, a union bound followed by Markov’s inequality yields

[TABLE]

In addition,

[TABLE]

Theorem 5.5**.**

Let $P\subset\ell_{1}^{d}$ such that $|P|=n$. For any $\varepsilon\in(0,1/2)$, there is a non-linear randomized embedding $h^{\prime}:\ell_{1}^{d}\rightarrow\ell_{1}^{k}$, where $k=\left(\log{\lambda_{P}}\cdot\log(d/\varepsilon)\right)^{\Theta(1/\varepsilon)}/\zeta(\varepsilon)$, for a function $\zeta(\varepsilon)>0$ depending only on $\varepsilon$, such that for any $q\in\ell_{1}^{d}$ , if there exists $p^{*}\in P$ such that $\|p^{*}-q\|_{1}\leq 1$, then with probability $\Omega(\varepsilon)$,

[TABLE]

Any point can be embedded in time $O(dk)$.

Proof 5.6**.**

We follow the same reasoning as in the proof of Theorem 4.3. The embedding is $h^{\prime}=f\circ g_{\varepsilon/d}$, where $f$ is the randomized linear map defined in Section 4. As before, we apply $h^{\prime}$ to every point in $P$, and only $f$ to queries. The randomly shifted grid incurs an additive error of $\varepsilon$ in the distances between $q$ and $P$.

Assume wlog that $q=0^{d}$ and let $A_{i}$ be the points of $g_{\varepsilon/d}(P)$ within distance $D_{i+1}=2^{2(i+1)}D_{0}$ from $q$. Hence, by Lemma 5.3,

[TABLE]

As in Lemma 4.1, for $D_{0}=\left\lceil{800T}/{k}\right\rceil=\Theta(\log{(k/\varepsilon)})$, $k\geq 20\log{\lambda_{P}}\cdot\log\left(\frac{dD_{0}}{\varepsilon\delta}\right)$ and $\delta=\varepsilon/5$,

[TABLE]

Hence, for $k=\Omega\left((\log^{2}{\lambda_{P}}\cdot\log(d/\varepsilon)\right)$, with probability at least $1-\varepsilon/5$, we have:

[TABLE]

Now, we are able to use Theorem 2.5 for points which are at distance at most $D_{0}+\varepsilon$ from $q$, and the near neighbor. By Lemma 5.3, with constant probability, the number of grid points at distance $\leq D_{0}+\varepsilon$, is at most $32\cdot\lambda_{P}^{4\log(dD_{0}/\varepsilon)}$. Hence, by Theorem 2.5, for $\gamma=\varepsilon/10$ and $\delta=\varepsilon/(160\lambda_{P}^{4\log{(dD_{0}/\varepsilon)}})$, with probability at least $1-\varepsilon/5$, it holds:

[TABLE]

Moreover, with probability at least $\varepsilon/2$, we obtain:

[TABLE]

As in Theorem 4.3, the target dimension needs to satisfy the following:

[TABLE]

Hence, for $k=\left(\log{\lambda_{P}}\cdot\log(d/\varepsilon)\right)^{\Theta(1/\varepsilon)}/\zeta(\varepsilon)$ we achieve total probability of success $\Omega(\varepsilon)$.

6 Conclusion

We have filled in a gap in the spectrum of randomized embeddings with bounded distortion only for distances between the query and a pointset: such embeddings existed for $\ell_{2}$ and $\ell_{1}$ and for doubling subsets of $\ell_{2}$. Here we settle the case of doubling subsets of $\ell_{1}$ with a near neighbor-preserving embedding. In the meantime, we obtain concentration bounds on sums of independent Cauchy variables. Our algorithms are quite simple, therefore they should also be of practical interest.

We rely on approximate $r$-nets or randomly shifted grids. For the former, Theorem 4.3 provides with a trade-off between the preprocessing time required and the target dimension. On the other hand, Theorem 5.5 has the advantage of fast preprocessing: any point is embedded in $O(dk)$ time, and the embedding is oblivious to the pointset. In regards to the near-linear preprocessing time, the two results are comparable, since the dimension in Theorem 5.5 can be substituted by the target dimension of Theorem 2.5.

Notice that any potential improvements to Theorem 2.5 should lead to improvements to Theorems 4.3 and 5.5. The target dimension in these theorems follows from a direct application of Theorem 2.5 to the representative data points which lie inside a bounding ball centered at the query.