Computational Concentration of Measure: Optimal Bounds, Reductions, and More

TL;DR

This paper presents a polynomial-time algorithm for computationally realizing measure concentration in product spaces, with applications to learning vulnerabilities, Gaussian distributions, and coin-tossing protocols.

Contribution

It introduces MUCIO, an algorithm for computational measure concentration, and establishes a reduction framework, extending results to weighted Hamming, Gaussian, and discrete processes.

Findings

Polynomial-time algorithm for measure concentration in product spaces.

Applications to cryptographic vulnerabilities and learning attacks.

Extensions to weighted Hamming, Gaussian, and coin-tossing protocols.

Abstract

Product measures of dimension $n$ are known to be concentrated in Hamming distance: for any set $S$ in the product space of probability $\epsilon$, a random point in the space, with probability $1-\delta$, has a neighbor in $S$ that is different from the original point in only $O(\sqrt{n\ln(1/(\epsilon\delta))})$ coordinates. We obtain the tight computational version of this result, showing how given a random point and access to an $S$-membership oracle, we can find such a close point in polynomial time. This resolves an open question of [Mahloujifar and Mahmoody, ALT 2019]. As corollaries, we obtain polynomial-time poisoning and (in certain settings) evasion attacks against learning algorithms when the original vulnerabilities have any cryptographically non-negligible probability. We call our algorithm MUCIO ("MUltiplicative Conditional Influence Optimizer") since proceeding through the coordinates, it decides to change each coordinate of the given point based on a multiplicative version of the influence of that coordinate, where influence is computed conditioned on previously updated coordinates. We also define a new notion of algorithmic reduction between computational concentration of measure in different metric probability spaces. As an application, we get computational concentration of measure for high-dimensional Gaussian distributions under the $\ell_1$ metric. We prove several extensions to the results above: (1) Our computational concentration result is also true when the Hamming distance is weighted. (2) We obtain an algorithmic version of concentration around mean, more specifically, McDiarmid's inequality. (3) Our result generalizes to discrete random processes, and this leads to new tampering algorithms for collective coin tossing protocols. (4) We prove exponential lower bounds on the average running time of non-adaptive query algorithms.

Equations

$$\hat{f}(w_1,\dots ,w_i)=\underset{(w_{i+1},\dots ,w_n)\leftarrow ({\mathbf{w}}_{i+1},\dots ,{\mathbf{w}}_n)}{E}[f(w_1,\dots ,w_n)].$$

$$\hat{f}(v_{\le n})-\hat{f}(\varnothing )=f(\overline{v})-1/2=\sum _{i\in [n]}\hat{g}(v_{\le i}).$$

$$\sum _{i=1}^n\ln \left(\frac{\hat{f}(v_{\le i})}{\hat{f}(v_{\le i-1})}\right)=\ln \left(\frac{\hat{f}(v_{\le n})}{\hat{f}(\varnothing )}\right)=\ln \left(\frac{1}{\hat{f}(\varnothing )}\right).$$

$$\ln (1/\hat{f}(v_{\le i-1}))-\underset{v_i\leftarrow \mathbf{v}[v_{\le i-1}]}{E}[\ln (1/\hat{f}(v_{\le i}))]\ge -\frac{{\lambda }^2}{2}.$$

$$\lambda \cdot E[\#\text{ of tampering}].$$

$$\lambda \cdot E[\#\text{ of tampering}]-\frac{n\cdot {\lambda }^2}{2}\le \ln (1/\epsilon ).$$

$$-\lambda \le \ln (\hat{f}(v_{\le i})/\hat{f}(v_{\le i-1}))\le \lambda .$$

$$E[\ln (\hat{f}(v_{\le i})/\hat{f}(v_{\le i-1}))]\ge -\frac{{\lambda }^2}{2}.$$

$$\mathrm{Pr}[\mathsf{Tam}(w_{\le i})\in \mathrm{Supp}(\mathbf{w}[w_{\le i-1}])]=1.$$

$$\underset{(\overline{u},\overline{v})\leftarrow ⟨\overline{\mathbf{w}}\Vert \mathsf{Tam}⟩}{\mathrm{Pr}}[\mathsf{d}(\overline{u},\overline{v})\le b]=1.$$

$$\underset{(\overline{u},\overline{v})\leftarrow ⟨\overline{\mathbf{w}}\Vert \mathsf{Tam}⟩}{E}[\mathsf{d}(\overline{u},\overline{v})]\le b.$$

$$E\left[e^{\mathbf{y}}\right]\le e^{{\lambda }^2/2}.$$

$$E[e^{-\min (\ln (\mathbf{x}),\lambda )}]\cdot e^s\le e^{{\lambda }^2/2}$$

$$E[\max (1/\mathbf{x},e^{-\lambda })]\le e^{{\lambda }^2/2}\cdot e^{-s}\le \frac{e^{{\lambda }^2}}{E[\mathbf{x}]-\delta \cdot c}.$$

$$\mathrm{Pr}\left[\sum _{i=1}^n{\mathbf{t}}_i\le -s\right]\le {\mathrm{e}}^{\frac{-(s-\gamma {)}^2}{2{\sum }_{i=1}^n{c}_i^2}}+n\cdot \xi .$$

$${\mathsf{HD}}_{\overline{\alpha }}(\overline{u},\overline{v})=\sum _{i\in [n],u_i\ne v_i}{\alpha }_i.$$

$$\ln (1/\tilde{\epsilon })\ge E[{\mathbf{K}}_{\overline{\alpha }}]\cdot \lambda -{\lambda }^{2}n/2+n\cdot \ln (1-3\gamma ).$$

$$\ln (1/\tilde{f}(v_{\le i-1}))-\underset{v_i\leftarrow \mathbf{v}[v_{\le i-1}]}{E}\left[\ln \left(1/\tilde{f}(v_{\le i})\right)\right]\ge \mathrm{Pr}[{\mathbf{k}}_i]\cdot ({\alpha }_i\lambda )-\frac{{\alpha }_i^2{\lambda }^2}{2}+\ln (1-3\gamma ).$$

$$\mathrm{Pr}[f({\mathbf{v}}_{\le n})=1]=1.$$

$$\ln (1/\tilde{\epsilon })$$

$$=\ln (1/\tilde{f}(\varnothing ))-E[\ln (1/\tilde{f}({\mathbf{v}}_{\le n}))]$$

$$=\sum _{i\in [n]}\left[E[\ln (1/\tilde{f}({\mathbf{v}}_{\le i-1}))-E[\ln (1/\tilde{f}({\mathbf{v}}_{\le i}))]\right]$$

$$\ge \sum _{i\in [n]}\left[E[{\alpha }_i\cdot {\mathbf{k}}_i]\cdot \lambda -\frac{{\alpha }_i^2\cdot {\lambda }^2}{2}+\ln (1-3\gamma )\right]$$

$$=E[{\mathbf{K}}_{\overline{\alpha }}]\cdot \lambda -\frac{n\cdot {\lambda }^2}{2}+\ln (1-3\gamma )\cdot n.$$

$$\ln (1/\tilde{f}(v_{\le i-1}))-\underset{v_i\leftarrow (\mathbf{v}[v_{\le i-1}]}{E}[\ln (1/\tilde{f}(v_{\le i}))]\ge \ln (1/\tilde{f}(v_{\le i-1}))-\ln (1/\widetilde{f^{\ast }})\ge \lambda {\alpha }_i$$

$$\underset{(u_i,v_i)\leftarrow (\mathbf{w},\mathbf{v})[v_{\le i-1}]}{\mathrm{Pr}}\left[\tilde{f}(v_{\le i})\ge \max \left(e^{-\lambda {\alpha }_i}\cdot \tilde{f}(v_{\le i-1}),\tilde{f}(v_{\le i-1},u_i)\right)\cdot e^{\lambda {\alpha }_i\cdot I(v_{\le i-1},u_i)}\right]=1.$$

$$\tilde{f}(v_{\le i})=\widetilde{f^{\ast }}\ge \tilde{f}(v_{\le i-1})\ge \tilde{f}(v_{\le i-1},u_i)\cdot e^{\lambda \cdot {\alpha }_i}$$

$$\tilde{f}(v_{\le i})=\tilde{f}(v_{\le i-1},u_i).$$

$$\underset{v_i\leftarrow \mathbf{v}[v_{\le i-1}]}{E}[\ln (\tilde{f}(v_{\le i}))]$$

$$\ge \underset{u_i\leftarrow \mathbf{w}[v_{\le i-1}]}{E}\left[\ln \left(\max \left(e^{-\lambda {\alpha }_i}\cdot \tilde{f}(v_{\le i-1}),\tilde{f}(v_{\le i-1},u_i)\right)\cdot e^{\lambda \cdot {\alpha }_i\cdot I(v_{\le i-1},u_i)}\right)\right]$$

$$=\underset{u_i\leftarrow \mathbf{w}[v_{\le i-1}]}{E}\left[\ln \left(\max \left(e^{-\lambda {\alpha }_i}\cdot \tilde{f}(v_{\le i-1}),\tilde{f}(v_{\le i-1},u_i)\right)\right)\right]+\lambda \cdot {\alpha }_i\cdot \underset{u_i\leftarrow \mathbf{w}[v_{\le i-1}]}{E}[I(v_{\le i-1},u_i)]$$

$$=\underset{u_i\leftarrow \mathbf{w}[v_{\le i-1}]}{E}\left[\ln \left(\max \left(e^{-\lambda {\alpha }_i}\cdot \tilde{f}(v_{\le i-1}),\tilde{f}(v_{\le i-1},u_i)\right)\right)\right]+\lambda \cdot {\alpha }_i\cdot E[{\mathbf{k}}_i].$$

$$\mathbf{t}\equiv \frac{\max \left(e^{-\lambda }\cdot \tilde{f}(v_{\le i-1}),\tilde{f}(v_{\le i-1},\mathbf{w}[v_{\le i-1}])\right)}{\tilde{f}(v_{\le i-1})}.$$

$$\mathrm{Pr}\left[e^{-\lambda {\alpha }_i}\le \mathbf{t}\right]=1.$$

Full text

We call our algorithm MUCIO (short for “MUltiplicative Conditional Influence Optimizer”) since proceeding through the coordinates of the product space, it decides to change each coordinate of the given point based on a multiplicative version of the influence of a variable, where the influence is computed conditioned on the value of all previously updated coordinates. MUCIO is an online algorithm in that it decides on the $i$’th coordinate of the output given only the first $i$ coordinates of the input. It also does not make any convexity assumption about the set ${\mathcal{S}}$.

Motivated by obtaining algorithmic variants of measure concentration in other metric probability spaces, we define a new notion of algorithmic reduction between computational concentration of measure in different probability metric spaces. This notion, whose definition has some subtlety, requires two (inverse) algorithmic mappings one of which is an algorithmic Lipschitz mapping and the other one is an algorithmic coupling connecting the two distributions. As an application, we apply this notion of reduction to obtain computational concentration of measure for high-dimensional Gaussian distributions under the $\ell_{1}$ distance.

We further prove several extensions to the results above as follows. (1) Generalizing in another dimension, our computational concentration result is also true when the Hamming distance is weighted. (2) As measure concentration is usually proved for concentration around mean, we show how to use our results above to obtain algorithmic concentration for that setting as well. In particular, we prove a computational variant of McDiarmid’s inequality, when properly defined. (3) Our result generalizes to discrete random processes (instead of just product distributions), and this generalization leads to new tampering algorithms for collective coin tossing protocols. (4) Finally, we prove exponential lower bounds on the average running time of non-adaptive query algorithms for proving computational concentration for the case of product spaces. Perhaps surprisingly, such lower bound shows any efficient algorithm must query about ${\mathcal{S}}$-membership of points that are not close to the original point even though we are only interested in finding a close point in ${\mathcal{S}}$.

1. 1 Introduction

2. 1.1 Our Results

3. 1.1.1 Extensions

4. 1.1.2 Polynomial-time Attacks on Robust Learning

5. 1.2 Technical Overview

6. 1.2.1 Our Approach (MUCIO: MUltiplicative Conditional Influence Optimizer)

7. 1.2.2 More Computational Concentration Results through Algorithmic Reductions

8. 1.2.3 Lower Bounds for Simple Methods

9. 2 Preliminaries

10. 3 Optimal Computational Concentration for Hamming Distance

11. 3.1 Proof Using Promised Approximate Partial Expectation Oracles

12. 3.1.1 Tampering with Abort

13. 3.2 Putting Things Together

14. 4 Algorithmic Reductions for Computational Concentration

15. 4.1 Case of Gaussian or Sphere under $\ell_{2}$

16. 5 Computational Concentration around Mean

17. 6 Limits of Nonadaptive Methods for Proving Computational Concentration

18. 7 Acknowledgement

1 Introduction

Let $({\mathcal{X}},{{\mathsf{d}}},{\bm{\upmu}})$ be a metric probability space in which ${{\mathsf{d}}}$ is a metric over ${\mathcal{X}}$, and ${\bm{\upmu}}$ is a probability measure over ${\mathcal{X}}$. The concentration of measure phenomenon [Led01, MS86] states that many natural metric probability spaces of high dimension are concentrated in the following sense. Any set ${\mathcal{S}}\subseteq{\mathcal{X}}$ of “not too small” probability ${\bm{\upmu}}({\mathcal{S}})\geq\varepsilon$ is “close” (according to ${{\mathsf{d}}}$) to “almost all” points ( $1-\delta$ measure according to ${\bm{\upmu}}$).

A well-studied class of concentrated spaces is the set of product spaces in which the measure ${\bm{\upmu}}={\bm{\upmu}}_{1}\times\dots\bm{\upmu}_{n}$ is a product measure of dimension $n$, and the metric ${{\mathsf{d}}}$ is Hamming distance of dimension $n$; namely, $\mathsf{HD}(\overline{u},\overline{v})=|\left\{i\colon u_{i}\neq v_{i}\right\}|$ for vectors $\overline{u}=(u_{1},\dots,u_{n}),\overline{v}=(v_{1},\dots,v_{n})$. More specifically, it is known, e.g., by results implicit in [AM80, MS86] and explicit in [McD89, Tal95], and weaker versions known as blowing-up lemma proved in [AGK76, Mar74, Mar86], that any such metric probability space is a so-called Normal Lévy family [Lév51, AM85]. Namely, for any ${\mathcal{S}}$ of probability ${\bm{\upmu}}({\mathcal{S}})\geq\varepsilon$, at least $1-\delta$ fraction of the points (under the product measure ${\bm{\upmu}}$) are $O(\sqrt{n\cdot\ln(\nicefrac{{1}}{{\varepsilon\delta}})})$-close in Hamming distance to ${\mathcal{S}}$. Previous proofs of measure concentration, and in particular those proofs for product spaces are information theoretic, and only show the existence of a “close” such point $\overline{y}\in{\mathcal{S}}$ to most of $\overline{x}\leftarrow{\bm{\upmu}}$ sampled according to ${\bm{\upmu}}$. Naive sampling of points around $\overline{x}$ will likely not fall into ${\mathcal{S}}$ (see Section 6).

Motivated by finding polynomial-time attacks on the “robustness” of machine learning algorithms, recently Mahloujifar and Mahmoody [MM19] studied a computational variant of the measure concentration in which the mapping from a given point $\overline{x}\leftarrow{\bm{\upmu}}$ to its close neighbor $\overline{y}\in{\mathcal{S}}$ is supposed to be computed by an efficient polynomial-time algorithm $A^{{\mathcal{S}},{\bm{\upmu}}}(\overline{x})=\overline{y}$ that has oracle access to test membership in ${\mathcal{S}}$ and a sampling oracle from the measure ${\bm{\upmu}}$.111In case of product measure, oracle access to a sampler from ${\bm{\upmu}}={\bm{\upmu}}_{1}\times\dots\bm{\upmu}_{n}$ is equivalent to having such samplers for all ${\bm{\upmu}}_{i}$. It was shown in [MM19] that if ${\mathcal{S}}$ is large enough, then the measure computationally concentrates around ${\mathcal{S}}$. In particular, it was shown that if $\Pr[{\mathcal{S}}]\geq 1/\operatorname{polylog}(n)$, then $A^{{\mathcal{S}},{\bm{\upmu}}}(\overline{x})$ finds $\overline{y}$ with Hamming distance $\widetilde{O}(\sqrt{n})$ from $\overline{x}$, and instead if ${\mathcal{S}}$ is at least $\Pr[{\mathcal{S}}]\geq\omega(1/\sqrt{n})$, then $A$ finds $\overline{y}$ with Hamming distance $o(n)$. Consequently, it was left open to prove computational concentration of measure around any smaller sets of “non-negligible” ${1}/{\operatorname{poly}(n)}$ probability, e.g., of measure $1/n$.

1.1 Our Results

In this work, we resolve the open question about the computational concentration of measure in product spaces under Hamming distance and prove (tight up to constant) computational concentration for all range of initial probabilities $\Pr[{\mathcal{S}}]$ for the target set ${\mathcal{S}}$. Namely, we prove the following result matching what information theoretic concentration of product spaces guarantees up to a constant factor, while the mapping is done algorithmically. As we deal with algorithms, without loss of generality, we focus on discrete distributions.222Note that even seemingly non-discrete distributions like Gaussian, when used as input to efficient algorithms, are necessarily rounded to limited precision and thus end up being discrete.

Theorem 1.1** (Main result).**

There is an algorithm $A_{\varepsilon,\delta}^{{\mathcal{S}},{\bm{\upmu}}}(\cdot)$ called MUCIO (short for “MUltiplicative Conditional Influence Optimizer”) that given access to a membership oracle for any set ${\mathcal{S}}$ and a sampling oracle from any product measure ${\bm{\upmu}}$ of dimension $n$, it achieves the following. If $\Pr[{\mathcal{S}}]\geq\varepsilon$, given $\varepsilon$ and $\delta$, the algorithm $A_{\varepsilon,\delta}^{{\mathcal{S}},{\bm{\upmu}}}(\cdot)$ runs in time $\operatorname{poly}(\nicefrac{{n}}{{\varepsilon\delta}})$, and with probability $\geq 1-\delta$ given a random point $\overline{x}\leftarrow{\bm{\upmu}}$, it maps $\overline{x}$ to a point $\overline{y}\in{\mathcal{S}}$ of bounded Hamming distance $\mathsf{HD}(\overline{x},\overline{y})\leq O(\sqrt{n\cdot\ln(\nicefrac{{1}}{{\varepsilon\delta}})})$.

See Theorem 3.2 for a more general version of Theorem 1.1.

For the special case that $\varepsilon,\delta={1}/{\operatorname{poly}(n)}$ (implying ${\mathcal{S}}$ has a non-negligible measure) the algorithm MUCIO of Theorem 1.1 achieves its goal in $\operatorname{poly}(n)$ time, while it changes only $\widetilde{O}(\sqrt{n})$ of the coordinates.

Our work can be seen as another example of works in computer science that make previously existential proofs algorithmic. A good example of a similar successful effort is the active line of work started from [Mos09, MT10] that presented algorithmic proofs of Lovász’s local lemma, leading to algorithms that efficiently find objects that previously where only shown to exist using Lovász’s local lemma. The work of [IK10] also approaches measure concentration from an algorithmic perspective, but their goal is to algorithmically find witness for lack of concentration.

1.1.1 Extensions

In this work we also prove several extensions to our main result in different directions expanding a direct study of computational concentration as an independent direction.

Extension to random processes and coin-tossing attacks. We prove a more general result than Theorem 1.1 in which the perturbed object is a random process. Namely, suppose $\overline{\mathbf{w}}\equiv(\mathbf{w}_{1},\dots,\mathbf{w}_{n})$ is a discrete (non-product) random process in which, given the history of blocks $w_{1},\dots,w_{i-1}$, the $i^{\mathrm{th}}$ block $w_{i}$ is sampled from its corresponding random variable $(\mathbf{w}_{i}\mid w_{1},\dots,w_{i-1})$. Suppose $\Pr_{\overline{w}\leftarrow\mathbf{w}}[\overline{w}\in{\mathcal{S}}]\geq\varepsilon$ for an arbitrary set ${\mathcal{S}}$. A natural question is: how much can an adversary increase the probability of falling into ${\mathcal{S}}$, if it is allowed to partially tamper with the online process of sampling $w_{1},\dots,w_{n}$ up to $K<n$ times? In other words, the adversary has a limited budget of $K$, and in the $i^{\mathrm{th}}$ step, it can use one of its budget, and in exchange it gets to override the originally (honestly) sampled value $w_{i}\leftarrow(\mathbf{w}_{i}\mid w_{1},\dots,w_{i-1})$ by a new value. Note that if the adversary does a tampering, the changed value will substitute $w_{i}$ and will affect the way the future blocks of the random process are sampled, e.g., in the next sampling of $w_{i+1}\leftarrow(\mathbf{w}_{i+1}\mid w_{1},\dots,w_{i})$.

Our generalized version of Theorem 1.1 (stated in Theorem 3.2) shows that in the above setting of tampering with random processes, an adversary with budget $O(\sqrt{n\cdot\ln(\nicefrac{{1}}{{\varepsilon\delta}}}))$ can indeed change the distribution of the random process and make the resulting tampered sequence end up in ${\mathcal{S}}$ with probability at least $1-\delta$, while the adversary also runs in time $\operatorname{poly}(\nicefrac{{n}}{{\varepsilon\delta}})$. Previously, [MM19] also showed a similar less tight result for random processes, but their result was limited to the setting that ${\mathcal{S}}$ is sufficiently large $\Pr[{\mathcal{S}}]\geq\omega(1/\sqrt{n})$.

The variant of Theorem 1.1 for random processes allows us to attack cryptographic coin-tossing protocols [BOL89, CI93, MPS10, BHT14, HO14, KKR18] in which $n$ parties $P_{1},\dots,P_{n}$ each send a single message during a total of $n$ rounds, and the full transcript $M=(m_{1},\dots,m_{n})$ determines a bit $b$. The goal of an attacker is to corrupt up to $K$ of the parties and bias the bit $b$ towards its favor. Our results show that even if the original bit $b$ had a small probability of being $1$, $\Pr_{\text{no-attack}}[b=1]\geq\varepsilon=1/\operatorname{poly}(n)$, then a $\operatorname{poly}(n)$-time attacker who can corrupt up to $\widetilde{O}(\sqrt{n})$ parties and change their messages can bias the output bit $b$ all the way up to make it $\Pr_{\text{attack}}[b=1]\geq 1-1/\operatorname{poly}(n)$. The corruption model here was first introduced by Goldwasser, Kalai and Park [GKP15] and is called strong adaptive corruption, because the adversary has the option to first see the message $m_{i}$ before deciding to corrupt (or not corrupt) $P_{i}$ to change its message $m_{i}$ (or not). 333If each message $m_{i}$ is a bit, it turns out that our attack can be modified to an attack that is not strong.

Weighted Hamming distance. In another extension to our Theorem 1.1 (see Theorem 3.2) we allow the Hamming distance to have different costs $\alpha_{i}$ when changing the $i^{\mathrm{th}}$ coordinate for any vector ${\overline{\alpha}}=(\alpha_{1},\dots,\alpha_{n})$ of $\ell_{2}$ norm $\sum_{i}\alpha_{i}^{2}=n$. In Talagrand’s inequality [Tal95], it is proved that even if ${\overline{\alpha}}_{\overline{x}}$ can completely depend on the original point $\overline{x}$, we still can conclude that most points are “close” to any sufficiently large set ${\mathcal{S}}$, when the distance from $\overline{x}$ to ${\mathcal{S}}$ is measured by the ${\overline{\alpha}}_{\overline{x}}$-weighted Hamming distance. An algorithmic version of Talagrand’s inequality, then, shall find a close point $\overline{y}\in{\mathcal{S}}$ to $\overline{x}$ measured by ${\overline{\alpha}}_{\overline{x}}$-weighed Hamming distance. Interestingly, our proof allows the coordinate $\alpha_{i}$ to completely depend on $(x_{1},\dots,x_{i-1})$, but falls short of proving an algorithmic version of Talagrand’s inequality, if possible at all.

Reductions and other metric probability spaces. Motivated by proving computational concentration of measure in other metric probability spaces, as well as designing a machinery for this goal, we define a new model of algorithmic reductions between computational concentration of measure in different metric probability spaces. This notion, whose definition has some subtle algorithmic aspects, requires two (inverse) polynomial-time mappings one of which is an algorithmic Lipschitz mapping and the other one is an algorithmic coupling connecting the two distributions. As an application, we apply this notion of reduction to obtain computational concentration of measure for high-dimensional Gaussian distributions under the $\ell_{1}$ distance. We prove this exemplary case by revisiting the proof of [B*+*97] who proved the information theoretic reduction from the concentration of Gaussian distributions under the $\ell_{1}$ distance to that of Hamming cube. We show how the core ideas of [B*+*97] could be extended to obtain all the algorithmic components that are needed for a computational variant. Although there are known results on concentration of Gaussian distribution $\ell_{1}$ in information theoretic regime, this is the first time (to the best of our knowledge) that a computational variant of concentration is proved for Gaussian spaces. We envision the same machinery can be applied to more information theoretic results for obtaining new computational variants; we leave doing so for future work. See Theorem 4.2 for the formal statement.

Computational concentration around mean. As measure concentration is usually proved for concentration around mean of a function $f(\cdot)$ when the inputs come from certain distributions, we show how to use our main result of Theorem 3.2 to obtain algorithmic concentration results for that setting as well. Namely, at a high level, we show that in certain settings (where concentration is known to follow from those settings) one can algorithmically find the right minimal perturbations to sampled points $\overline{x}$ so that the new perturbed point $\overline{x}^{\prime}$ gives us the average of the concentrated function: $f(\overline{x}^{\prime})\approx\operatorname*{\mathbb{E}}_{x\leftarrow{\bm{\upmu}}}[f(\overline{x})]$. Sometimes doing so is trivial (e.g., in case of Chernoff bound, when $f$ is simply the addition of i.i.d. sampled Boolean values, as one can greedily change Boolean variables to decrease their summation) but sometimes doing so is not straightforward. In particular, we prove a computational variant of McDiarmid’s inequality. Namely, we show how to modify $\sqrt{n}$ coordinates of a vector $\overline{x}\leftarrow{\bm{\upmu}}$ sampled from a product distribution ${\bm{\upmu}}$ of dimension $n$, such that $f(\overline{x}^{\prime})$ gets arbitrary (i.e., $1/\operatorname{poly}(n)$) close to the average $\mu=\operatorname*{\mathbb{E}}_{\overline{x}\leftarrow{\bm{\upmu}}}[f(\overline{x})]$ for a function $f$ that is Lipschitz under Hamming distance. (Note that the Lipschitz property is needed for the McDiarmid inequality as well). See Theorem 5.1 for the formal statement.

Lower bounds for simple methods. We also prove exponential lower bounds on the query complexity of natural, yet restricted, classes of algorithms. Two such classes stand out: One is non-adaptive algorithms where the queries made do not depend on the answer of previous queries. Another, natural class of algorithms are algorithms where all the queried points are at the distance where an acceptable final output may be at that distance. These lower bounds shed light on why perhaps some of the ideas behind our algorithm MUCIO are necessary, and that some simpler more straightforward algorithms are not as efficient.

Polynomial-time biasing attacks against extractors. At a high level, our biasing attacks on random processes are also related to impossibility results on extracting randomness from blockwise Santha-Vazirani sources [SV86, CG88, BEG17, RVW04, DOPS04] and specifically the $p$-tampering and $p$-resetting attacks of [BGZ16, MM17, MDM18]. In those attacks, an attacker might get to tamper each incoming block with an independent probability $p$, and they can achieve a bias of magnitude $O(p)$ (in polynomial time). However, our attackers can choose which blocks are the target of their tampering substitutions, but then achieve much stronger bias and almost fixing the output with much smaller $o(n)$ number of tamperings.

1.1.2 Polynomial-time Attacks on Robust Learning

Our results also have implications on (limits) of robust learning, which is also the focus of the work of [MM19] where computational concentration of measure was also studied. We refer the reader to [MM19] for a more in-depth treatment of the literature and settings for (attacks on) robust learning. For sake of completeness, below we describe the basic setting of such attacks and briefly discuss the implication of our computational concentration results to robust learning attacks.

Suppose $L$ is a (deterministic) learning algorithm, taking as input a training set $T$ consisting of $m$ iid sampled and labeled examples $T=\left\{x_{i},c(x_{i})\right\}_{i\in[m]}$ where $x_{i}\leftarrow{\bm{\upmu}}$ for $i\in[m]$, and that $c(\cdot)$ is a concept function to be learned. Let $h=L(T)$ be the hypothesis that the learner produces based on the training set $T$. Main attacks against robustness of learners are studied during the training phase or the testing phase of a learning process. We describe the settings and previous work before explaining the implication of our new computational concentration results to those settings.

Poisoning attacks.

In a so-called data poisoning attack [BNS*+*06, BNL12], which is tightly related to Valiant’s malicious noise model [Val85, KL93, BEK02], the adversary only tampers with the training phase and substitutes a small $p<1$ fraction of the examples in $T$ with other arbitrary examples, leading to a poisoned data set $\widetilde{T}$. The goal of the adversary, in general, is to make $L(\widetilde{T})$ produce a “bad” hypothesis $h\in\widetilde{H}$ (e.g., bad might mean having large risk or making a mistake on a particular test $x$ during the test time) where $\widetilde{H}\subseteq H$ includes the set of all undesired hypothesis. It was shown by [MDM19] that the concentration of measure in product spaces (under Hamming distance) implies that in any such learning process, so long as $\Pr_{T}[L(T)\in\widetilde{H}]\geq\varepsilon$, then an adversary $\mathsf{A}$ who changes $O(\sqrt{m\cdot\ln(\nicefrac{{1}}{{\varepsilon\delta}})})$ of the training examples (and substitute them with still correctly labeled data) can increase the probability of producing a bad hypothesis in $\widetilde{H}$ to $\Pr_{\widetilde{T}\leftarrow\mathsf{A}(T)}[L(\widetilde{T})\in\widetilde{H}]\geq\delta$. It was left open whether such attack can be made polynomial time, or that perhaps computational intractability can be leveraged to prevent such attacks. The work of [MM19] showed how to make such attacks polynomial time, only for the setting where the probability of falling into $\widetilde{H}$ was already not too small, and in particular at least $\omega(1/\sqrt{n})$, and also with looser bounds. Our Theorem 1.1 shows how to get such polynomial time evasion attacks for any non-negligible probability $\varepsilon\geq 1/\operatorname{poly}(n)$. In fact, as stated in Theorem 1.1, our attack’s complexity can gracefully adapt to $\varepsilon$.

The previous attacks of [MDM19, MM19] and our newer attacks of this work do not contradict recent exciting works in defending against poisoning attacks [DKK*+*16, LRV16, DKK*+*18, PSBR18], as those defenses either focus on learning parameters of distributions or, even in the classification setting, they aim to bound the risk of the hypothesis, while we increase the probability of a bad Boolean property.444In fact, the challenge in those works is to obtain polynomial-time learners in settings where inefficient robust methods were perhaps known in the robust statistics literature. The focus here, however, is to obtain polynomial-time attacks.

Evasion attacks.

In another active line of work, other types of attacks on learners are studied in which the adversary enters the game during the test time. In such so-called evasion attacks [BFR14, CW17, SZS*+*14, GMP18] that find “adversarial examples”, the goal of the adversary is to perturb the test input $x$ into a “close” input $\widetilde{x}$ under some metric ${{\mathsf{d}}}$ (perhaps because this small perturbation is imperceptible to humans) in such a way that this tampering makes the hypothesis $h$ make a mistake. In [MDM19], it was also shown that the concentration of measure can potentially lead to inherent evasion attacks, as long as the input metric probability space $({\mathcal{X}},{{\mathsf{d}}},{\bm{\upmu}})$ is concentrated. This holds e.g., if the space is a Normal Lévy family [Lév51, AM85]. The work of [MM19] showed the existence of polynomial time evasion attacks with sublinear perturbations for classification tasks in which the input distribution is a $n$-dimensional product space (e.g., the uniform distribution over the hypercube) under Hamming distance. But their attacks could be applied only when the original risk $\varepsilon$ of the hypothesis $h$ is at least $\varepsilon=\omega(1/\sqrt{n})$. However, standard PAC learners (e.g., based on empirical risk minimization) can indeed achieve polynomially small risk $\varepsilon=1/\operatorname{poly}(m)$ where $m$ is the sample complexity. Our Theorem 1.1 shows how to obtain polynomial-time attacks even in the low-risk regime $\varepsilon=1/\operatorname{poly}(n)$555Note that in the “high dimensional” setting where input dimension $n$ is huge, we can see the sample complexity $m$ bounded, which implies $\varepsilon\geq 1/\operatorname{poly}(m)$ if $\varepsilon=1/\operatorname{poly}(n)$. and perturb given samples $x\leftarrow{\bm{\upmu}}$ in $\widetilde{O}(\sqrt{n})$ coordinates and make the perturbed adversarial instance $\widetilde{x}$ misclassified with high probability.

Our results of Section 4 show that one can also obtain polynomial time evasion attacks for classifiers whose inputs come from metric probability spaces that use metrics other than Hamming distance (e.g., Gaussian under $\ell_{1}$). Using the reductionist approach of Section 4 one can perhaps obtain more such results. Our attacks, however, do not rule out the possibility of robust classifiers for specific input distributions such as images or voice that is the subject of recent intense research [SZS*+*14, CW17, MFF16], but they shed light on barriers for robustness in theoretically natural settings. See [BPR18, DV19] for more discussion on other possible barriers for robust learning.

1.2 Technical Overview

In this subsection, we describe the challenges and key ideas behind the proof of Theorem 1.1 and some of its extensions. The extension for the concentration around mean (see Section 5) follows directly from the main result about concentration around noticeably large sets. Thus, we only focus on explaining ideas behind some other extensions to our result; namely how to obtain new results through carefully defined algorithmic reductions, and proving limits for the power of simple methods for proving computational concentration.

Setting. (The reader might find the explanations for our notation at the beginning of Section 2 useful.) Suppose $\overline{\mathbf{w}}\equiv(\mathbf{w}_{1}\times\dots\times\mathbf{w}_{n})$ is a random variable with a product distribution of dimension $n$.666As discussed above, our results extend to random processes as well, when formalized carefully, but for simplicity we focus on the interesting special case of product distributions. Also, suppose the set ${\mathcal{S}}\subseteq\operatorname{Supp}(\mathbf{w})$ is denoted by its characteristic function $f$, where $f(\overline{w})=1$ iff $\overline{w}\in{\mathcal{S}}$. The goal of the tampering algorithm $\mathsf{Tam}$ is to change as few as possible of the sampled blocks $\overline{w}=(w_{1},\dots,w_{n})\leftarrow\overline{\mathbf{w}}$ making the new vector $\overline{v}=(v_{1},\dots,v_{n})$ such that $f(\overline{v})=1$ with high probability (over the both steps of sampling $\overline{w}$ and obtaining $\overline{v}$ from it).

Our starting point is the previous attack of [MM19] that only proved computational concentration around large sets of measure $\Pr[{\mathcal{S}}]\geq\omega(1/\sqrt{n})$. The result of [MM19], in turn, was built upon techniques developed in the work of Komargodski, Raz, and Kalai [KKR18] that presented an alternative simpler proof for a previously known result of Lichtenstein et al. [LLS89]. Below, we first describe the high level ideas behind the approach of [MM19, KKR18], and then we describe why that approach breaks down when ${\mathcal{S}}$ gets smaller than $1/\sqrt{n}$, and thus fails to obtain the optimal information theoretic bounds for concentration. We then describe our new techniques to bypass this challenge and obtain computational concentration with optimal bounds.

The high-level approach of [MM19].

As it turns out, the tampering algorithm of [MM19], as well as ours, do not need to know $w_{i+1},\dots,w_{n}$ when deciding to change $w_{i}$ (into a different $v_{i}\neq w_{i}$) or leaving it as is (i.e., $w_{i}=v_{i}$). So, a useful notation to use is the partial expected values, capturing the chance of falling into ${\mathcal{S}}$ (i.e., $f(\overline{w})=1$) over the randomness of the remaining blocks.

[TABLE]

One obvious reason for working with $\hat{f}(\cdot)$ quantities is that they can be approximated with arbitrary small $\pm 1/\operatorname{poly}(n)$ additive error. This can be done using the sampling oracle of the distribution of $\overline{\mathbf{w}}\equiv\mathbf{w}_{1}\times\dots\times\mathbf{w}_{n}$ and the oracle $f(\cdot)$ determining membership in ${\mathcal{S}}$.

At a high level, the idea behind the attack of [MM19] is to change $w_{i}$ only if this change allows us to increase $\hat{f}(\cdot)$ additively by $+\lambda$ for a parameter $\lambda\approx 1/\sqrt{n}$. We first describe this attack, and then explain its challenges against obtaining optimal bounds and how we resolve them.

At a high level, the attack of [MM19] tampers with the $i^{\mathrm{th}}$ block (i.e., $w_{i}$), if just before or just after looking at $w_{i}$, we conclude that we can increase $\hat{f}(\cdot)$ by $\lambda$.

Construction 1.2** (Attack of [MM19] oracle $\hat{f}(\cdot)$).**

Suppose that we are given a prefix ${v}_{\leq i-1}$ that is finalized, and we are also given a candidate value $w_{i}$ for the $i$’th block (supposedly sampled from $\mathbf{w}_{i}$) and we want to decide to keep it $v_{i}=w_{i}$ or change it $v_{i}\neq w_{i}$. Let $\lambda>0$ be a parameter of the attack to be chosen later, $v^{*}_{i}=\operatorname*{argmax}_{y_{i}}\hat{f}({v}_{\leq i-1},y_{i})$ be the choice for $i$’th block that maximizes $\hat{f}({v}_{\leq i})$, and let $f^{*}=\hat{f}({v}_{\leq i-1},v^{*}_{i})$.

(Case 1) If f^{*}\geq\hat{f}({v}_{\leq i-1}){\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}+\lambda}, then output $v_{i}=v_{i}^{*}$ (regardless of $w_{i}$). 2. 2.

(Case 2) Otherwise, if (by looking at $w_{i}$) \hat{f}({v}_{\leq i-1},w_{i})\leq\hat{f}({v}_{\leq i-1}){\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}-\lambda}, then again output $v_{i}=v_{i}^{*}$. 3. 3.

(Case 3) Otherwise, keep the value $w_{i}$ and output $v_{i}=w_{i}$.

Why this attack biases $f(\cdot)$ towards 1? For simplicity, support $\Pr[{\mathcal{S}}]=1/2$. Suppose we “color” different $i\in[n]$ depending on whether the tampering algorithm changes the $i^{\mathrm{th}}$ block $w_{i}$ or not. If $v_{i}\neq w_{i}$ (tampering happened), color $i$ green, denoted by $i\in G$, and otherwise color $i$ red, denoted as $i\in R=[n]\setminus G$. A simple yet extremely useful observation is that we can write $f(\overline{v})$ as the sum of the changes in $\hat{f}(v_{\leq i})$ between consecutive $i$. Namely, if we let $\hat{g}(v_{\leq i})=\hat{f}(v_{\leq i})-\hat{f}(v_{\leq i-1}),$ then

[TABLE]

This means that we have to study the affect of the green and red coordinates $i$ on how $\hat{g}(v_{\leq i})$ behaves, because that will tell us how the final output bit is determined and distributed.

Construction 1.2 is designed so that, whenever $i$ is green, the partial expectation oracle $\hat{f}(v_{\leq i})$ jumps up at least by $\lambda$ (i.e., $\hat{g}(v_{\leq i})\geq\lambda$). So, the only damage (leading to falling outside ${\mathcal{S}}$) could come from the red coordinates and how they change $\hat{f}(v_{\leq i})$ downwards. Let us now focus on the red coordinates $i\in R$. A simple inspection of Construction 1.2 shows that, the change in $\hat{f}(\cdot)$ captured by $\hat{g}(v_{\leq i})$ is bounded in absolute value by $\lambda$, and that is the result of no-tampering for a block. Therefore, the summation of $\hat{g}(v_{\leq i})$ for red coordinates $i$ would cancel out each other and, by the Azuma inequality, the probability that this summation is more than $1$ is at most $\exp(-1/(n\cdot\lambda^{2})$. So, by choosing $\lambda\ll 1/\sqrt{n}$, the red coordinates cannot control the final bit, as with high probability this summation is less than one. This means that the outcome (whenever the red coordinates do not fix the function) should be $1$, because the green coordinates only increase the $\hat{f}(\cdot)$ function.

Why the attack is efficient? The efficiency of the attack follows form its effectiveness and the same argument described above. Namely, whenever the green coordinates are determining the output, it means that their total sum of of $\hat{g}(v_{\leq i})$ is going from a specific number in $[0,+1]$ to $1$, and each time they jump up by at least $\lambda$, so they cannot be more than $n/\lambda$ green steps. Since we chose $\lambda=1/\sqrt{n}$, the efficiency follows as well.

The challenge when $\Pr[{\mathcal{S}}]=\operatorname*{\mathbb{E}}[\overline{w}]=\varepsilon$ is too small.

The issue with the above approach is that whenever $\varepsilon$ is too small (not around $1/2$) we need to pick $\lambda$ much smaller, so that the summation (i.e., the effect of the red coordinates does not make the function reach zero). Simple calculation shows that after the threshold $\varepsilon\approx 1/\sqrt{n}$, the number of tampered (green) blocks would grow too much and eventually become more than $n$. However, note that when we reach $n$ tamperings, it means the attack’s efficiency is meaningless.

1.2.1 Our Approach (MUCIO: MUltiplicative Conditional Influence Optimizer)

Main step 1: tampering with multiplicatively influential blocks.

Our first key idea is to judge whether a block is influential (and thus tamper it) based how much it can change the partial expectations in a multiplicative way. (This is related to the notion of a log-likelihood ratio in statistics and information theory.) Construction 1.3 below describes this simple change. However, as we will see, doing this simple change will have big advantages as well as new challenges to be resolved. We will describe both the advantages and thew new challenges after the construction.

Construction 1.3** (Multiplicative online tampering using oracle $\hat{f}(\cdot)$).**

The key difference between this attack and that of Construction 1.2 is that here, in order to judge whether tampering with the current $i^{\mathrm{th}}$ block is worth it or not, we make the decision based on the multiplicative gain (in how $\hat{f}(\cdot)$ changes) that this would give us. Namely, for the same setting of Construction 1.2, we do as follows.

(Case 1) If f^{*}\geq{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}e^{\lambda}\cdot}\hat{f}({v}_{\leq i-1}), then output $v_{i}=v_{i}^{*}$ (regardless of $w_{i}$). 2. 2.

(Case 2) Otherwise, if \hat{f}({v}_{\leq i-1},w_{i})\leq{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}e^{-\lambda}\cdot}\hat{f}({v}_{\leq i-1}), then output $v_{i}=v_{i}^{*}$. 3. 3.

(Case 3) Otherwise, keep the value $w_{i}$ and output $v_{i}=w_{i}$.

Main advantage: the output is fully biased. We first describe what advantages the above change gives us, and then will discuss the remaining challenges. The key insight into why this is a better approach is that the tampering algorithm of Construction 1.3 will always lead to obtaining $f(\overline{v})=1$ at the end (i.e., we always end up in ${\mathcal{S}}$). In order to see why this is a big difference, notice that if $\hat{f}({w}_{\leq 0})=\varepsilon$ is very small at the beginning and we tamper only based on additive differences (as is done in Construction 1.2), there is a possibility that we do not tamper with the first block and end up at $\hat{f}({w}_{\leq 1})=0$. Such a problem does not happen when we decide on tampering based on multiplicative improvement, and every tiny chance of falling into ${\mathcal{S}}$ is taken advantage of.

Only few tamperings happen. To analyze the number of tamperings that occur in the “idealized” attack of Construction 1.3 we keep track of $\ln\left({\hat{f}(v_{\leq i})}/{\hat{f}(v_{\leq i-1})}\right)$ as we go. We know that the output of function under the attack is always 1 which means:

[TABLE]

We again categorize the indices $i$ to red and green. Green set indicates the locations that the algorithm tampers with $w_{i}$ and red is the set of locations that tampering has not happened and $v_{i}=w_{i}$. For the red locations, we prove the following inequality that plays a key role in our analysis of the attack. One interpretation of this inequality is that we will now use $\ln(1/\hat{f}(v_{\leq i-1}))$ as a potential function that allows us keep track of, and control, the number of tamperings.

[TABLE]

This inequality follows from a Jensen Gap inequality on the natural logarithm function. For green locations, we increase $\ln(\hat{f}(v_{\leq i}))$ whenever we tamper by at least $\lambda$. Therefore, the overall effect of green locations on $\sum_{i=1}^{n}\ln({\hat{f}(v_{\leq i})}/{\hat{f}(v_{\leq i-1})})$ will be

[TABLE]

Combining these together we get the following:

[TABLE]

Now we can optimize $\lambda$ to get the best inequality on the expected number of tampering.

New challenge: obtaining good multiplicative approximations when $\hat{f}(\cdot)$ gets too small. Construction 1.3 increases the average to 1 (i.e., we always end up in ${\mathcal{S}}$) with small number of tampering. However we cannot implement that construction in polynomial time. The problem is that it is hard to instantiate the oracle $\hat{f}(v_{\leq i})$ polynomial time when the partial average gets close to [math]. To solve this issue, we add a step to the construction that makes the algorithm abort if the partial average goes below some threshold.

Construction 1.4** (Online tampering with abort $\mathsf{TamAb}$ using partial-expectations oracle).**

This construction is identical to Construction 1.3, except that whenever the fixed prefix has a too small partial expectation $\hat{f}({v}_{\leq i-1},w_{i})$ (based on a new parameter $\tau$) we will abort. Also, in that case the tampering algorithm does not tamper with any future $v_{i}$ block either. Namely, we add the following “Case 0” to the previous steps:

• •

(Case 0) If $\hat{f}({v}_{\leq i-1},w_{i})\leq e^{-\tau}\cdot\varepsilon$ abort. If had aborted before, do nothing.

Main step 2: showing that reaching low expectations is unlikely under the attack.

To argue that the new construction does not hurt the performance of our algorithm by much, we show that the probability of getting a low $\hat{f}(v_{\leq i})$ is small because of the way our algorithm works. The idea is that, our algorithm always guarantees that

[TABLE]

We also show that

[TABLE]

This means that the sequence of $\ln\left({\frac{\hat{f}(v_{\leq i})}{\hat{f}(v_{\leq i-1})}}\right)$ forms an “approximate” sub-martingale difference sequence. We can use Azuma inequality to show that sum of this sequence will remain bigger than some small threshold, with high probability. After all, we can bound the probability of getting into Case 0 to be very small.

1.2.2 More Computational Concentration Results through Algorithmic Reductions

Here we explain a technical overview of our generic reduction technique. Let $S_{1}=({\mathcal{X}}_{1},{{\mathsf{d}}}_{1},{\bm{\upmu}}_{1})$ and $S_{2}=({\mathcal{X}}_{2},{{\mathsf{d}}}_{2},{\bm{\upmu}}_{2})$ be two metric probability spaces. In addition, assume we already know some level of computational concentration proved for $S_{2}$, and that we want to prove (some level of) computational concentration for $S_{1}$ through a reduction. In Section 4, we formalize a generic framework to prove such reductions. The main ingredients of such algorithmic reduction are two polynomial time mappings $\mathbf{f}\colon{\mathcal{X}}_{1}\to{\mathcal{X}}_{2}$ and $\mathbf{g}\colon{\mathcal{X}}_{2}\to{\mathcal{X}}_{1}$ with 3 properties. The first property (roughly speaking) requires that $\mathbf{f}({\bm{\upmu}}_{1})\approx{\bm{\upmu}}_{2}$ and $\mathbf{g}({\bm{\upmu}}_{2})\approx{\bm{\upmu}}_{1}$. This property guarantees that if we sample a point from one space and use the mapping and go to the other space, we get a distribution close to the probability measure of the second space. (This can be interpreted as an algorithmic coupling.) The second property requires that the mapping $\mathbf{g}$ is Lipschitz. The third property requires that $\mathbf{g}(\mathbf{f}(x))$ is close to $x$. The idea behind why such reduction (as a collection of these mappings) work is as follows. We are given a point $x_{1}$ in $S_{1}$ and we want to find a close $x_{2}$ such that $x_{2}$ falls inside a subset ${\mathcal{S}}$. To do that we first map $x_{1}$ to a point $x^{\prime}_{1}$ in $S_{2}$ using $\mathbf{f}$. We know that $S_{2}$ is computationally concentrated and we can efficiently find a close $x^{\prime}_{2}$ such that $x^{\prime}_{2}$ falls into an specific subset ${\mathcal{S}}^{\prime}$. Then we use $\mathbf{g}$ to go back to a point $x_{2}$ in $S_{1}$. The second and third properties together guarantee that $x_{1}$ and $x_{2}$ are close, because $x^{\prime}_{1}$ and $x^{\prime}_{2}$ are close. At the same time, the first condition guarantees that $x_{2}$ will hit ${\mathcal{S}}$ if we select ${\mathcal{S}}^{\prime}$ in a careful way. See Theorem 4.2 for more details.

We use this general framework to prove computational concentration bounds for Gaussian spaces under $\ell_{1}$ norm. We reduce the computational concentration of Gaussian distribution under $\ell_{1}$ to the computational concentration of the Boolean Hamming cube. For this goal, we show how to build two mappings $\mathbf{f}$ and $\mathbf{g}$ from an $n$-dimensional Gaussian space to a $n^{2}$-dimensional Hamming cube and vice versa, following the footsteps of a reduction by [B*+*97] who proved an information theoretic variant of this result. Here we show that the algorithmic ingredients that are necessary, in addition to the ideas already in [B*+*97], could indeed be obtained. The main idea behind this mappings is the fact that the number of $1$’s in a sample from $n$-dimensional hamming cube approximately forms a Gaussian distribution centered around $\frac{n}{2}$. Therefore, we can map each dimension of the Gauss space to a $n$-dimensional hamming cube and vice versa. Here we observe that we can use the same idea and build the mappings in a way that achieves the three properties mentioned above. See Section 4 for more details.

1.2.3 Lower Bounds for Simple Methods

To prove exponential lower bounds on the query complexity of too-simple algorithms, we consider the half-space ${\mathcal{S}}$ in the Hamming cube consisting of those points with below-average Hamming weight.

A uniformly random point $\overline{x}$ in the cube, with high probability has Hamming distance $\Omega(\sqrt{n})$ from the set ${\mathcal{S}}$. Now, if for such a point $\overline{x}$, we hope to find a close point in ${\mathcal{S}}$ simply by sampling uniformly at random among points close to $x$, we fail except with exponentially small probability. For only random points with distance $n^{1-o(1)}$ have a significant chance of changing the weight of point $\overline{x}$ by $\Omega(\sqrt{n})$, whereas the information-theoretic bound says there exists a point of distance $O(\sqrt{n})$ that changes the weight by $\Omega(\sqrt{n})$.

To achieve lower bounds for more general classes of algorithms, we use a random half-space instead of a fixed half-space. This gives us exponential lower bounds for non-adaptive attacks as well as attacks that query about ${\mathcal{S}}$-membership of points outside a ball of size $d=O(\sqrt{n\cdot\ln(\nicefrac{{1}}{{\varepsilon\delta}})})$ even when we are interested in finding a point in the intersection of ${\mathcal{S}}$ and this ball. Notice that MUCIO avoids this last restriction by surveying the influence of the first coordinate on the totality of points, while it ends up changing only a small fraction of the coordinates.

2 Preliminaries

General notation.

We use calligraphic letters (e.g., ${\mathcal{X}}$) for sets. By default, all distributions and random variables in this work are discrete. We use bold letters (e.g., $\mathbf{w}$) to denote random variables that return a sample from a corresponding discrete distribution. By $w\leftarrow\mathbf{w}$ we denote sampling $w$ from the random variable $\mathbf{w}$. By $\operatorname{Supp}(\mathbf{w})$ we denote the support set of $\mathbf{u}$. For an event ${\mathcal{S}}\subseteq\operatorname{Supp}(\mathbf{w})$, the probability function of $\mathbf{w}$ for ${\mathcal{S}}$ is denoted as $\Pr[\mathbf{w}\in{\mathcal{S}}]=\Pr_{w\leftarrow\mathbf{w}}[w\in{\mathcal{S}}]$. For a randomized algorithm $R(\cdot)$, by $y\leftarrow R(x)$ we denote the randomized execution of $R$ on input $x$ outputting $y$. By $\mathbf{u}\equiv\mathbf{v}$ we denote that the random variables $\mathbf{u}$ and $\mathbf{v}$ have the same marginal distributions. Unless stated otherwise, we denote vectors by using a bar over a variable. By $\overline{\mathbf{w}}\equiv(\mathbf{w}_{1},\mathbf{w}_{2},\dots,\mathbf{w}_{n})$ we refer to a sequence of $n$ jointly sampled random variables. For a vector $\overline{w}=(w_{1}\dots w_{n})$, we use ${w}_{\leq i}$ to denote the prefix $(w_{1},\dots,w_{i})$, and we use the same notation ${\mathbf{w}}_{\leq i}$ for jointly distributed random variables. For a jointly distributed random variables $(\mathbf{u},\mathbf{v})$, by $(\mathbf{u}\mid v)$ we denote the conditional distribution $(\mathbf{u}\mid\mathbf{v}=v)$. For a random variable $\mathbf{u}$, by $T^{\mathbf{u}}(\cdot)$ we denote an oracle-aided algorithm $T^{(\cdot)}(\cdot)$ that can query fresh sample from $\mathbf{u}$. By $\mathbf{u}\times\mathbf{v}$ we refer to the product distribution in which $\mathbf{u}$ and $\mathbf{v}$ are sampled independently. For a real-valued random variable $\mathbf{x}$, by $\operatorname*{\mathbb{E}}[\mathbf{x}]$ we refer to the expected value of $\mathbf{x}$, and by $\operatorname*{\mathbb{V}}[\mathbf{x}]$ we denote its variance.

Notation on random processes and online samplers.

Let $\overline{\mathbf{w}}\equiv(\mathbf{w}_{1},\dots,\mathbf{w}_{n})$ be a sequence of jointly distributed random variables. We can interpret the distribution of $\overline{\mathbf{w}}$ as a random process in which the $i^{\mathrm{th}}$ block $w_{i}$ is sampled from the marginal distribution $(\mathbf{w}_{i}\mid{w}_{\leq i-1})$. For simplicity, we use notation $\mathbf{w}[{w}_{\leq i-1}]\equiv(\mathbf{w}_{i}\mid{w}_{\leq i-1})$ to refer to this marginal conditional distribution. (Note that $i$ is dropped from the distribution’s name, relying on the input ${w}_{\leq i-1}$ that uniquely determines $i$.) We can interpret ${w}_{\leq i-1}$ as a “node” in a tree of depth $i$, and the sampling $w_{i}\leftarrow\mathbf{w}[{w}_{\leq i-1}]$ can be seen as the process of sampling the next child according to the distribution of $\mathbf{w}[{w}_{\leq i-1}]$. Alternatively, describing the distributions of the random variables $\mathbf{w}[{w}_{\leq i-1}]$ defines the distribution of $\overline{\mathbf{w}}$. For random variable $\overline{\mathbf{w}}\equiv(\mathbf{w}_{1},\dots,\mathbf{w}_{n})$ we sometimes refer to the random variable $\mathbf{w}[{w}_{\leq i-1}]$ as the online sampler for $\overline{\mathbf{w}}$, because it returns fresh samples form the next block, given the previously fixed prefix ${w}_{\leq i-1}$.

Definition 2.1** (Online tampering).**

Let $\overline{\mathbf{w}}\equiv(\mathbf{w}_{1},\dots,\mathbf{w}_{n})$ be a sequence of jointly distributed random variables, and let $\mathbf{w}[{w}_{\leq i-1}]$ be the online sampler for $\overline{\mathbf{w}}$ for all $i\in[n]$ and all ${w}_{\leq i-1}\in\operatorname{Supp}({\mathbf{w}}_{\leq i-1})$. Online tampering algorithms for $\overline{\mathbf{w}}$ and their properties are defined as follows.

• •

Online tampering. We call a (potentially randomized and computationally unbounded) algorithm $\mathsf{Tam}$ an online tampering algorithm for $\overline{\mathbf{w}}$, if for all $i\in[n]$ and ${w}_{\leq i}\in\operatorname{Supp}({\mathbf{w}}_{\leq i})$, it holds that

[TABLE]

Namely, $\mathsf{Tam}({w}_{\leq i})$ always outputs a candidate $i^{\mathrm{th}}$ block that still falls into $\operatorname{Supp}(\mathbf{w}[{w}_{\leq i-1}])$.

• •

Resulting tampered distribution. For an online tampering algorithm $\mathsf{Tam}$ for $\overline{\mathbf{w}}$, by $(\overline{\mathbf{u}},\overline{\mathbf{v}})\equiv\langle\overline{\mathbf{w}}\,\,\|\,{\mathsf{Tam}}\rangle$ we refer to the jointly distributed sequence of random varaibles defined as follows. For $i=1,2,\dots,n$, we first sample $u_{i}\leftarrow\mathbf{w}[{v}_{\leq i-1}]$, and then we obtain $v_{i}\leftarrow\mathsf{Tam}({v}_{\leq i-1},u_{i})$ as the (possibly different than $u_{i}$) choice of the tampering algorithm $\mathsf{Tam}$ for the $i^{\mathrm{th}}$ block (that will override $u_{i}$). At the end, we output the pair of sequences $(\overline{u}={u}_{\leq n},\overline{v}={v}_{\leq n})$ as the sample from $(\overline{\mathbf{u}},\overline{\mathbf{v}})$.

Notation. For simplicity, we use $\mathbf{v}[{v}_{\leq i-1}]$ to denote $(\mathbf{v}_{i}\mid{v}_{\leq i-1})$ and use $(\mathbf{w},\mathbf{v})[{v}_{\leq i-1}]$ to denote the jointly distributed random variables from which $(u_{i},v_{i})$ are sampled conditioned on the prefix ${v}_{\leq i-1}$. The notation allows us to use $\mathbf{v}[{v}_{\leq i-1}],(\mathbf{w},\mathbf{v})[{v}_{\leq i-1}]$ similarly to how we use online samplers.777Note that are not defining a similar notation of the form $\mathbf{u}[{v}_{\leq i-1}]$ for $\overline{\mathbf{u}}$. Firstly, this is not needed as $\mathbf{w}[{v}_{\leq i-1}]$ already provides a sampler for $u_{i}$. Moreover, such notation would be inconsistent with our notation for online samplers for random processes based on joint distributions, because the notation would implicitly interpret ${v}_{\leq i-1}$ as previous samples from ${\mathbf{u}}_{\leq i-1}$.

• •

Budget of tampering attacks. Let ${{\mathsf{d}}}$ be a metric defined over $\operatorname{Supp}(\overline{\mathbf{u}})$ as vectors of dimension $n$. We say a tampering algorithm $\mathsf{Tam}$ has budget (at most) $b$, if

[TABLE]

We say that $\mathsf{Tam}$ has average budget (at most) $b$, if the following weaker condition holds

[TABLE]

• •

Algorithmic efficiency of attacks. If $\overline{\mathbf{w}}=\overline{\mathbf{w}}_{n}$ is a member from a family defined for all $n\in{\mathbb{N}}$, we call an online or offline tampering algorithm efficient, if its running time is $\operatorname{poly}(N)$ where $N$ is the total bit-length representation of any $\overline{w}\in\operatorname{Supp}(\overline{\mathbf{w}}_{n})$.

Definition 2.2** (Partial expectations).**

Suppose $f\colon\operatorname{Supp}(\overline{\mathbf{w}})\mapsto{\mathbb{R}}$ for $\overline{\mathbf{w}}\equiv(\mathbf{w}_{1},\dots,\mathbf{w}_{n})$, $i\in[n]$, and ${w}_{\leq i}\in\operatorname{Supp}({\mathbf{w}}_{\leq i})$. Then (using a small hat) we define the notation $\hat{f}(w_{\leq i})=\operatorname*{\mathbb{E}}_{\overline{w}\leftarrow({\overline{\mathbf{w}}}\mid{w}_{\leq i})}[f(\overline{w})]$ to define the expected value of $f$ for a sample from $\overline{\mathbf{w}}$ given the prefix ${w}_{\leq i}$. In particular, for $\overline{w}={w}_{\leq n}$, we have $\hat{f}(\overline{w})=f(\overline{w})$, and also $\hat{f}(\varnothing)=\operatorname*{\mathbb{E}}[f(\overline{\mathbf{w}})].$

Lemma 2.3** (Hoeffding’s lemma).**

Let $\mathbf{x}$ be a random variable such that $\Pr[a\leq\mathbf{x}\leq b]=1$ and $\operatorname*{\mathbb{E}}\left[\mathbf{x}\right]=0$. Then, it holds that $\operatorname*{\mathbb{E}}[e^{\mathbf{x}}]\leq e^{(b-a)^{2}/8}$.

Lemma 2.4**.**

Let $\mathbf{x}$ be a random variable where $\Pr\left[e^{-\lambda}\leq\mathbf{x}\right]=1$ and $\Pr\left[\mathbf{x}\leq e^{\lambda}\right]\geq 1-\delta$ and $\Pr\left[\mathbf{x}\leq c\right]=1$. Then, $\operatorname*{\mathbb{E}}\left[\ln(\mathbf{x})\right]\geq\ln(\operatorname*{\mathbb{E}}\left[\mathbf{x}\right]-\delta\cdot c)-\lambda^{2}/2$.

Proof.

Let $\operatorname*{\mathbb{E}}\left[\min(\ln(\mathbf{x}),\lambda)\right]=s$. Consider a random variable $\mathbf{y}\equiv\min(\ln(\mathbf{x}),\lambda)-s$. We have $\operatorname*{\mathbb{E}}[\mathbf{y}]=0$ and $-\lambda-s\leq\mathbf{y}\leq\lambda-s$. Therefore, by Lemma 2.3 we have

[TABLE]

On the other hand, we have $\operatorname*{\mathbb{E}}\left[e^{\mathbf{y}}\right]=\operatorname*{\mathbb{E}}\left[e^{\min(\ln(\mathbf{x}),\lambda)-s}\right]=\operatorname*{\mathbb{E}}\left[\min\left(\mathbf{x},e^{\lambda}\right)\right]\cdot e^{-s}.$ Thus, we have $\operatorname*{\mathbb{E}}\left[\min\left(\mathbf{x},e^{\lambda}\right)\right]\cdot e^{-s}\leq e^{\lambda^{2}/2}$ which implies $e^{-s}\leq e^{\lambda^{2}/2-\ln\left(\operatorname*{\mathbb{E}}\left[\min\left(\mathbf{x},e^{\lambda}\right)\right]\right)}$, and so $s\geq\ln\left(\operatorname*{\mathbb{E}}\left[\min\left(\mathbf{x},e^{\lambda}\right)\right]\right)-\lambda^{2}/2$. Therefore we have, $s\geq\ln\left(\operatorname*{\mathbb{E}}\left[\mathbf{x}\right]-\delta\cdot c\right)-\lambda^{2}/2$. ∎

Lemma 2.5**.**

Let $\mathbf{x}$ be a random variable where $\Pr[e^{-\lambda}\leq\mathbf{x}]=1$ and $\Pr[\mathbf{x}\leq e^{\lambda}]\geq 1-\delta$ and $\Pr[\mathbf{x}\leq c]=1$. Then, $\operatorname*{\mathbb{E}}[1/\mathbf{x}]\leq\frac{e^{\lambda^{2}}}{\operatorname*{\mathbb{E}}[\mathbf{x}]-\delta\cdot c}$.

Proof.

Let $\operatorname*{\mathbb{E}}[\min(\ln(\mathbf{x}),\lambda)]=s$. Consider a random variable $\mathbf{y}=\min(\ln(\mathbf{x}),\lambda)-s$. Similar to proof of Lemma 2.4 we have $s\geq\ln(\operatorname*{\mathbb{E}}[\mathbf{x}]-\delta\cdot c)-\lambda^{2}/2$. Now consider another random variable $\mathbf{y}^{\prime}\equiv-\mathbf{y}$. Again by using Hoeffding Lemma we have $\operatorname*{\mathbb{E}}[e^{\mathbf{y}^{\prime}}]\leq e^{\lambda^{2}/2}$ which means

[TABLE]

which implies

[TABLE]

∎

The following lemma is implied by Theorem 3.13 from [MHRAR98].

Lemma 2.6** (Azuma’s inequality for sub-martingales).**

Let $\overline{\mathbf{t}}\equiv(\mathbf{t}_{1},\dots,\mathbf{t}_{n})$ be a sequence of $n$ jointly distributed random variables such that for all $i\in[n]$, $\Pr[|\mathbf{t}_{i}|\leq c_{i}]\geq 1-\xi$, for all ${t}_{\leq i-1}\leftarrow{\mathbf{t}}_{\leq i-1}$, and that $\operatorname*{\mathbb{E}}[\mathbf{t}_{i}\mid{t}_{\leq i-1}]\geq-\gamma_{i}.$ If $\gamma=\sum_{i=1}^{n}\gamma_{i}$, then we have

[TABLE]

3 Optimal Computational Concentration for Hamming Distance

In this section, we formally state and prove our main result, which is the computational concentration of measure in any product space under Hamming distance.

Definition 3.1** (Weighted Hamming Distance).**

For ${\overline{\alpha}}=(\alpha_{1},\dots,\alpha_{n})\in{\mathbb{R}}_{+}^{n}$, the ${\overline{\alpha}}$-weighted Hamming distance between vectors of dimension $n$ is denoted by $\mathsf{HD}_{{\overline{\alpha}}}(\cdot,\cdot)$ and is defined as

[TABLE]

Theorem 3.2**.**

Let $(\alpha_{1},\dots,\alpha_{n})\in{\mathbb{R}}^{n}$ be such that $\sum_{i=1}^{n}\alpha_{i}^{2}=n.$ Then, there is a (uniform) oracle-aided randomized algorithm $\mathsf{Tam}$ such that the following holds. Suppose $f\colon\operatorname{Supp}(\overline{\mathbf{w}})\mapsto\{0,1\}$ is a Boolean function for random variable $\overline{\mathbf{w}}\equiv(\mathbf{w}_{1},\dots,\mathbf{w}_{n})$, and that $\Pr[f(\overline{\mathbf{w}})=1]=\varepsilon$. Then, the oracle-aided algorithm $\mathsf{Tam}^{\mathbf{w}[\cdot],f(\cdot)}(\varepsilon,\delta,\cdot)$ (also denoted by $\mathsf{Tam}$ for simplicity) with access to the online sampler $\mathbf{w}[\cdot]$ for $\overline{\mathbf{w}}$ and $f(\cdot)$ as oracles is an online tampering algorithm for $\overline{\mathbf{w}}$ and has the following features:

$\Pr[f(\overline{\mathbf{v}})=1]\geq 1-\delta$* where $\overline{\mathbf{v}}$ is the tampered sequence, i.e., $\Pr_{(\overline{u},\overline{v})\leftarrow\langle\overline{\mathbf{w}}\,\,\|\,{\mathsf{Tam}}\rangle}[f(\overline{v})=1]\geq 1-\delta$.* 2. 2.

$\mathsf{Tam}$’s tampering budget in $\alpha$-weighed Hamming distance $\mathsf{HD}_{{\overline{\alpha}}}$ is $O(\sqrt{n\cdot\ln(\nicefrac{{1}}{{\varepsilon\delta}})})$. 3. 3.

$\mathsf{Tam}$* runs in time $\operatorname{poly}(\nicefrac{{N}}{{\varepsilon\delta}})$ where $N$ is the total bit representation of any $\overline{w}\leftarrow\overline{\mathbf{w}}$.*

Remark 3.3** (Corollary for product distributions).**

If the original random variable $\overline{\mathbf{w}}=(\mathbf{w}_{1},\dots,\mathbf{w}_{n})$ in Theorem 3.2 is a product, $\overline{\mathbf{w}}=(\mathbf{w}_{1}\times\dots\times\mathbf{w}_{n})$, then the distribution of the samples $\mathbf{u}$ obtained through $(\overline{u},\overline{v})\leftarrow\langle\overline{\mathbf{w}}\,\,\|\,{\mathsf{Tam}}\rangle$ would be identical to that of $\overline{\mathbf{w}}$. Namely, we can simply think of the samples $\overline{u}$ as the original untampered vector sampled from $\overline{\mathbf{w}}$, and $\overline{v}$ would be the perturbed vector.

In the rest of this section, we prove Theorem 3.2.

3.1 Proof Using Promised Approximate Partial Expectation Oracles

The following result works in the model where the approximate partial-expectations oracle $\tilde{f}(\cdot)$ is available to the online tampering algorithm $\mathsf{AppTam}$.

Consider three oracles $\tilde{f}(v_{\leq i})$ , $m(v_{\leq i})$ and $\tilde{f^{*}}(v_{\leq i})=\tilde{f}({v}_{\leq i},m(v_{\leq i}))$ with the guarantee that for all ${v}_{\leq i}\in\operatorname{Supp}(\mathbf{w}_{\leq i})$ we have 5 conditions:

$\left|\ln{\tilde{f}(v_{\leq i})}-\ln{\hat{f}(v_{\leq i})}\right|\leq\gamma$, 2. 2.

$\tilde{f^{*}}(v_{\leq i})=\tilde{f}({v}_{\leq i},m(v_{\leq i}))\geq\tilde{f}(v_{\leq i})$, 3. 3.

$\Pr\left[\tilde{f}({v}_{\leq i},\mathbf{w}[{v}_{\leq i}])\geq\tilde{f^{*}}(v_{\leq i})\right]\leq\gamma\cdot\tilde{f}(v_{\leq i})$, 4. 4.

$0\leq\tilde{f}(v_{\leq i})\leq 1$, 5. 5.

$\tilde{f}(v_{\leq n})=f({v}_{\leq n}).$

The first condition states that the approximate partial expectation oracle has a small multiplicative error. The second and third conditions state that $m(v_{\leq i-1})$ is a good approximation of some $v*$ that maximized $\tilde{f}({v}_{\leq i-1},v^{*})$. Now we construct an algorithm using these oracles.

Construction 3.4** (Online tampering using promised approximate partial-expectations oracle).**

Recall that we are given a prefix ${v}_{\leq i-1}$ that is finalized, and we are also given a candidate value $u_{i}$ for the $i$’th block (supposedly sampled from $\mathbf{w}[{v}_{\leq i-1}]$) and we want to decide to keep $v_{i}=u_{i}$ or change it. Let $\lambda>0$ be a parameter of the attack to be chosen later, $v^{*}_{i}=\tilde{f^{*}}(v_{\leq i-1})$ and let $\tilde{f^{*}}=\tilde{f^{*}}(v_{\leq i-1})$ be that maximum.

(Case 1) If $\tilde{f^{*}}\geq e^{\lambda\alpha_{i}}\cdot\tilde{f}({v}_{\leq i-1})$, then output $v_{i}=v_{i}^{*}$ (regardless of $u_{i}$). 2. 2.

(Case 2) Otherwise, if $\tilde{f}({v}_{\leq i-1},u_{i})\leq e^{-\lambda\alpha_{i}}\cdot\tilde{f}({v}_{\leq i-1})$ , then output $v_{i}=v_{i}^{*}$. 3. 3.

(Case 3) Otherwise keep the value $u_{i}$ and output $v_{i}=u_{i}$.

Claim 3.5** (Average case analysis of Construction 3.4).**

Let $\mathbf{k}_{i}$ be the Boolean random variable that $\mathbf{k}_{i}=1$ iff the tampering over the $i$’th block happens, and let ${\mathbf{K}_{\overline{\alpha}}}=\sum_{i\in[n]}\alpha_{i}\cdot\mathbf{k}_{i}$ capture the resulting $\mathsf{HD}_{{\overline{\alpha}}}$ distance between the jointly sampled $\overline{u}$ and $\overline{v}$. Also let $\tilde{\varepsilon}=\tilde{f}(\varnothing)$. Then, it holds that

[TABLE]

Corollary of Claim 3.5.

By choosing $\lambda=\sqrt{2\ln(1/\varepsilon)/n}$, we obtain $\operatorname*{\mathbb{E}}[{\mathbf{K}_{\overline{\alpha}}}]\leq\sqrt{2n\ln(1/\varepsilon)}$.

We prove the following stronger statement that implies Claim 3.5.

Claim 3.6**.**

Let ${v}_{\leq i-1}$ be fixed. Then,

[TABLE]

Proof of Claim 3.5 using Claim 3.6..

A key property of Construction 3.4 is that, because the tampering algorithm does not allow the function reach [math], the final sequence $\overline{v}$ always makes the function $1$, namely

[TABLE]

Using the above equation, Claim 3.5 follows from Claim 3.6 and linearity of expectation as follows.

[TABLE]

∎

Now we prove Claim 3.6.

Proof of Claim 3.6.

There are two cases:

• •

If tampering of Case 1 happens, then we have $\Pr[\mathbf{k}_{i}=1]=1$, and

[TABLE]

Thus, in this case Claim 3.6 follows trivially.

• •

If tampering of Case 1 does not happen, it means that $\tilde{f^{*}}$ is bounded from above. In the following, we focus on this case and all the probabilities and expectations are conditioned on Case 1 not happening; namely, we have $\tilde{f^{*}}\leq\tilde{f}(v_{\leq i-1})\cdot e^{\lambda}$ .

Let $I({v}_{\leq i})$ be the indicator function for the set $\left\{{v}_{\leq i}\colon{\tilde{f}(v_{\leq i})}\leq e^{-\lambda\alpha_{i}}\cdot{\tilde{f}(v_{\leq i-1})}\right\}.$ We have

[TABLE]

This is correct because we are either in Case 2, which means $I({v}_{\leq i})=1$ and

[TABLE]

or we are in Case 3 which means $I({v}_{\leq i})=0$ and

[TABLE]

Note that the two terms on each side of the inequality inside the probability above depend only on either of $u_{i}$ or $v_{i}$ (not both). Therefore, by linearity of expectation we have

[TABLE]

Now consider the random variable $\mathbf{t}$ for a fixed ${v}_{\leq i-1}$ as follows

[TABLE]

It holds that

[TABLE]

We also know by condition 3 of the $\tilde{f^{*}}(\cdot)$ oracle that

[TABLE]

which together with $\tilde{f^{*}}(v_{\leq i-1})\leq\tilde{f}(v_{\leq i-1})\cdot e^{\lambda\alpha_{i}}$ implies

[TABLE]

We also know that

[TABLE]

We also have

[TABLE]

which implies

[TABLE]

Therefore using 3, 4, 5 and 6 and applying Lemma 2.4 we get,

[TABLE]

Combining Equations (2) and (7), we get

[TABLE]

which finishes the proof. ∎

Claim 3.7** (Worst case analysis of Construction 3.4).**

Let $\mathbf{k}_{i}$ be the Boolean random variable that $\mathbf{k}_{i}=1$ iff the tampering over the $i$’th block happens, and let ${\mathbf{K}_{\overline{\alpha}}}=\sum_{i\in[n]}\alpha_{i}\cdot\mathbf{k}_{i}$ capture the resulting $\mathsf{HD}_{{\overline{\alpha}}}$ distance between the jointly sampled $\overline{u}$ and $\overline{v}$. Also let $\tilde{\varepsilon}=\tilde{f}(\varnothing)$. Then, it holds that

[TABLE]

Proof.

We prove this claim by induction on $n$. Let $A(n,k,\tilde{\varepsilon})$ be a function that indicates the maximum probability of using more than $k$ budget, over all random processes with boolean outcome of length $n$, and average $\tilde{\varepsilon}$. We want to inductively show that

[TABLE]

Consider different cases that might happen during the tampering of first block. If we tamper on first block through Case I, we have

[TABLE]

And by induction hypothesis we have

[TABLE]

So the induction goes through for Case 1. If we are not in Case 1, then we have,

[TABLE]

We know that $\operatorname*{\mathbb{E}}[\max(e^{-\lambda\cdot\alpha_{i}}\cdot\tilde{\varepsilon},\tilde{f}(u_{\leq 1})))]\geq\operatorname*{\mathbb{E}}[\tilde{f}(u_{\leq 1})]\geq\tilde{\varepsilon}\cdot e^{-\gamma}$. Now we can use Lemma 2.5 and get

[TABLE]

Combining Equations 8 and 9 we get,

[TABLE]

which finishes the proof. ∎

3.1.1 Tampering with Abort

The Construction 3.4 achieves average close to 1 with small number of tampering. However we cannot implement that construction it in polynomial time. The problem is that it is hard to instantiate the oracle $\tilde{f}(\cdot)$ and $\tilde{f^{*}}(\cdot)$ in polynomial time when the partial average gets close to [math]. Following we add a step to our construction to address this issue. Then we will show that this additional step will not hurt the performance of the algorithm by much.

Construction 3.8** (Online tampering with abort $\mathsf{AppTamAb}$ using promised approximate partial-expectations oracle).**

This construction is identical to Construction 3.4, except that whenever the fixed prefix has a too small approximate partial expectation $\tilde{f}({v}_{\leq i-1},u_{i})$ (based on a parameter $\tau$) we will abort. Also, in that case the tampering algorithm does not tamper with any future $v_{i}$ block either. Namely, we add the following “Case 0” to the previous steps:

• •

(Case 0) If $\tilde{f}({v}_{\leq i-1},u_{i})\leq e^{-\tau}\cdot\tilde{\varepsilon}$ abort ($\tilde{\varepsilon}=\tilde{f}(\varnothing)$). If had aborted before, do nothing.

Average and worst case analysis of Construction 3.8.

The average number of tampering of Construction 3.8 is trivially less than average number of tampering of Construction 3.4. Therefore, the same bound of Claim 3.5 still applies to Construction 3.8 as well. Also, the probability of number of tampering going beyond some threshold does not increase compared to Construction 3.4 which means the same bound of Claim 3.7 hold here.

Claim 3.9**.**

The probability of ever aborting during sampling $(\overline{u},\overline{v})\leftarrow\langle\overline{\mathbf{w}}\,\,\|\,{\mathsf{TamAb}}\rangle$ is at most $n\cdot e^{-\frac{(\tau-n\cdot\lambda^{2}/2)^{2}}{2\cdot n\cdot\lambda^{2}}}$. As a result, we also have

[TABLE]

Proof.

Define Boolean indicator functions $I_{0},I_{1},I_{2}$ and $I_{3}$, as well as a real-valued vector $\overline{y}$ as follows. The first function $I_{0}$ indicates that we have not aborted yet, and the others define a condition for their corresponding cases in Construction 3.4.

[TABLE]

[TABLE]

[TABLE]

The last function indicates that the above conditions are not happening.

[TABLE]

Finally, we define a real-valued function $y$ as follows that captures the change in the potential function for the cases where none of $I_{0},I_{1},I_{2}$ are happening.

[TABLE]

Now consider a sequence of random variables $\overline{\mathbf{y}}=(\mathbf{y}_{1},\dots,\mathbf{y}_{n})$ sampled as follows. We first sample $(\overline{u},\overline{v})\leftarrow(\overline{\mathbf{u}},\overline{\mathbf{v}})$ then set $y_{i}=y({v}_{\leq i-1},u_{i})=y({v}_{\leq i})$ for $i\in[n]$. Note that $y({v}_{\leq i-1},u_{i})=y({v}_{\leq i})$ because if $I_{3}({v}_{\leq i-1},u_{i})=1$ it means that $u_{i}=v_{i}$.

Claim 3.10**.**

We have $\operatorname*{\mathbb{E}}[e^{\mathbf{y}_{i}}\mid{y}_{\leq i-1}]\geq e^{-2\gamma}$.

Notation.

Since $I_{j}(\cdot)$’s are Boolean, we can use the notation $(I_{i}\lor I_{j})({v}_{\leq i})$ or $(1-(I_{i}\lor I_{j}))({v}_{\leq i})$ based on logical operators to construct more Boolean indicators.

Proof of Claim 3.10.

The high level idea is that $e^{\mathbf{y}_{i}}$ is approximately equal to $\hat{f}({v}_{\leq i-1},{u_{i}})/\hat{f}(v_{\leq i})$ when we are in Case 3. The average of $\hat{f}({v}_{\leq i-1},{u_{i}})/\hat{f}(v_{\leq i})$ conditioned on Case 2 and Case 3 is exactly 1. We know that in Case 2 the average is less than one, therefore the average in Case 3 should be at least 1. Following, we formalize this idea.

[TABLE]

∎

Claim 3.11**.**

We have $\Pr[\mathbf{y}_{i}\geq-\lambda\cdot\alpha_{i}]=1$ and $\Pr[\mathbf{y}_{i}\leq\lambda\cdot\alpha_{i}]\geq 1-\gamma\cdot\tilde{f}(v_{\leq i-1})$.

Proof.

If $I_{3}({v}_{\leq i})=0$ then $y({v}_{\leq i})=0$ and both inequalities hold. On the other hand, If $I_{3}({v}_{\leq i})=1$ it means that $e^{-\lambda\cdot\alpha_{i}}\cdot\tilde{f}(v_{\leq i-1})\leq\tilde{f}(v_{\leq i})$. Also $\Pr[\mathbf{y}_{i}\leq e^{\lambda\cdot\alpha_{i}}\cdot\tilde{f}(v_{\leq i-1})]\geq 1-\gamma\cdot\tilde{f}(v_{\leq i-1})$ holds because of gaurantee of the oracle $\tilde{f^{*}(\cdot)}$. ∎

Claim 3.12**.**

We have $\operatorname*{\mathbb{E}}[\mathbf{y}_{i}\mid{y}_{\leq i-1}]\geq\ln(1-3\gamma)-\frac{\lambda^{2}\cdot\alpha_{i}^{2}}{2}$.

Proof.

The proof follows by using Lemma 2.4 and Claims 3.10 and 3.11. ∎

Claim 3.13**.**

The probability of aborting is at most $n\cdot e^{\frac{(\tau-\lambda n\cdot\lambda^{2}/2)^{2}}{2\cdot n\cdot\lambda^{2}}}.$

Proof.

By Claims 3.12 and 3.11, the sequence $\overline{\mathbf{y}}=(\mathbf{y}_{1},\dots,\mathbf{y}_{n})$ forms an (approximate) submartingale and by Azuma inequality of Lemma 2.6 we have,

[TABLE]

On the other hand, for every ${v}_{\leq i}\in\operatorname{Supp}({\mathbf{v}}_{\leq i})$ we have $I_{2}({v}_{\leq i})=0.$ Therefore, for every ${v}_{\leq j}\in\operatorname{Supp}({\mathbf{v}}_{\leq j})$,

[TABLE]

We now calculate probability of the event $A_{j}$ that the partial average goes bellow $e^{-\tau}\cdot\tilde{\varepsilon}$ (i.e., abort happens) at the $j^{\mathrm{th}}$ block for the first time.

[TABLE]

The above means that the probability that the tampering algorithm of Construction 3.8 enters the abort state is less than $n\cdot e^{-\frac{(\tau-n\cdot\lambda^{2}/2)^{2}}{2\cdot n\cdot\lambda^{2}}}+n^{2}\cdot\gamma$. ∎

We already know that if abort does not happen then the output will always be $1$. Therefore, we have

[TABLE]

[TABLE]

∎

3.2 Putting Things Together

In this subsection we show how to instantiate parameters of Construction 3.8 so that we can get polynomial time attack. We first show how to instantiate the oracles. To compute oracle $\tilde{f}(v_{\leq i})$, we sample $\frac{8}{\gamma^{3}\cdot e^{-\tau}\cdot\tilde{\varepsilon}}$ random continuation and take the average over all of them. By Hoeffding inequality, if $\hat{f}(v_{\leq i})\geq e^{-\tau}\cdot\tilde{\varepsilon}$ we get the following:

[TABLE]

For $m(v_{\leq i})$ and $\tilde{f^{*}}(v_{\leq i})$ oracle, sample $\frac{1}{\gamma^{2}\cdot e^{-\tau}\cdot\tilde{\varepsilon}}$ number of $v_{i+1}$ and take the maximum over $\tilde{f}(v_{\leq i+1})$. This way, we can easily bound the probability of Conditions 2 or 3 not happening by $\gamma$ for all ${v}_{\leq i}$ that $\tilde{f}(v_{\leq i})\geq e^{-\tau}\cdot\tilde{\varepsilon}$. Note that in both of these oracle, we are ignoring the case where $\tilde{f}(v_{\leq i})$ is smaller than the threshold that causes the construction to abort. This enables us to achieve high confidence on our oracles. Using these oracles, we can bound the average of function, average budget and worst case budget of construction 3.8 as follows. Based on Claim 3.9 we have

[TABLE]

The last $-2n\cdot\gamma$ is added to the right hand side to capture the probability of any of the algorithm’s oracle calls failing. For the average budget, following Claim 3.5 we have,

[TABLE]

And for the worst case budget, following Claim 3.7 we have

[TABLE]

Instantiating the Average Case Algorithm:

Now if we set $\lambda=\sqrt{\nicefrac{{-2\ln(\varepsilon)}}{{n}}}$, $\tau=\ln(\nicefrac{{1}}{{\tilde{\varepsilon}}})+\sqrt{4\ln(\nicefrac{{\delta}}{{2n}})\cdot\ln(\tilde{\varepsilon})}$ and $\gamma=\frac{\delta}{24n^{2}}$ then we can provide the oracles in time $poly(\nicefrac{{n}}{{\varepsilon\cdot{\delta}}})$ and we get:

[TABLE]

and

[TABLE]

Instantiating the Worst Case Algorithm:

Also, for the worst case attacks. If we select the tampering budget $k=\sqrt{2n\cdot\ln(\delta/8)\cdot\ln(\varepsilon/2})$ and then let $\lambda=k/2n$. For $\tau=\ln(\nicefrac{{1}}{{\tilde{\varepsilon}}})+\sqrt{4\ln(\nicefrac{{\delta}}{{2n}})\cdot\ln(\tilde{\varepsilon})}$ and $\gamma=\min(\delta/24n^{2},\varepsilon/4n)$ we get an algorithm that runs in time $poly(\nicefrac{{n}}{{\varepsilon\cdot{\delta}}})$, uses at most $k$ tamperings and increases the average as follows

[TABLE]

4 Algorithmic Reductions for Computational Concentration

In this section, we show a generic framework to prove computational concentration for a metric probability space by reducing its computational concentration to that of another metric probability space. We first define an embedding with some properties.

Definition 4.1**.**

Let $S_{1}=({\mathcal{X}}_{1},{{\mathsf{d}}}_{1},{\bm{\upmu}}_{1})$ and $S_{2}=({\mathcal{X}}_{2},{{\mathsf{d}}}_{2},{\bm{\upmu}}_{2})$ be two metric probability spaces. We call a pair of mappings $(\mathbf{f},\mathbf{g})$ (where $\mathbf{f}$ and $\mathbf{g}$ are potentially randomized) an $(\alpha,b,w)$ computational concentration (CC) reduction from $S_{1}$ to $S_{2}$ if the following hold:

• •

Probability embedding. The distribution $\mathbf{f}({\bm{\upmu}}_{1})$ is $\alpha$-close (in statistical distance) to ${\bm{\upmu}}_{2}$ and $\mathbf{g}({\bm{\upmu}}_{2})$ is $\alpha$-close to ${\bm{\upmu}}_{1}$.

• •

Almost Lipschitz property of $\mathbf{g}$. With probability $1$ over all $x,x^{\prime}\leftarrow{\bm{\upmu}}_{2}$, ${{\mathsf{d}}}_{1}(\mathbf{g}(x),\mathbf{g}(x^{\prime}))\leq w\cdot{{\mathsf{d}}}_{2}(x,x^{\prime})+b$.

• •

Almost inverse mappings. For every $x_{1}\in{\mathcal{X}}_{1}$, and all $x_{2}\leftarrow\mathbf{f}(x_{1})$, it holds that ${{\mathsf{d}}}_{1}(x_{1},\mathbf{g}(x_{2}))\leq b$.

Now we have the following lemma which how to reduce computational concentration on a metric probability space by reducing it to computational concentration on another metric probability space using the embedding between them.

Theorem 4.2**.**

Let $S_{2}=({\mathcal{X}}_{2},{{\mathsf{d}}}_{2},{\bm{\upmu}}_{2})$ be a metric probability space and let $A_{2}^{{\mathcal{S}}(\cdot)}:{\mathcal{X}}_{2}\to{\mathcal{X}}_{2}$ be an oracle algorithm such that for any subset ${\mathcal{S}}\subseteq{\mathcal{X}}_{2}$ we have ${{\mathsf{d}}}_{2}(A_{2}^{{\mathcal{S}}(\cdot)}(x),x)\leq k$ and

[TABLE]

for a function $c\colon[0,1]\to[0,1]$. If $(\mathbf{f},\mathbf{g})$ is an $(\alpha,b,w)$ CC reduction from $S_{1}=({\mathcal{X}}_{1},{{\mathsf{d}}}_{1},{\bm{\upmu}}_{1})$ to $S_{2}=({\mathcal{X}}_{2},{{\mathsf{d}}}_{2},{\bm{\upmu}}_{2})$, then there is an oracle algorithm $A_{1}^{{\mathcal{S}}(\cdot)}\colon{\mathcal{X}}_{1}\to{\mathcal{X}}_{1}$ such that for any subset ${\mathcal{S}}\subseteq{\mathcal{X}}_{1}$ we have ${{\mathsf{d}}}_{1}(A_{1}^{{\mathcal{S}}(\cdot)}(x),x)\leq w\cdot k+2b$ and

[TABLE]

Furthermore, if $A_{2}$, $\mathbf{f}$ and $\mathbf{g}$ run in time $\operatorname{poly}(\frac{n}{\varepsilon})$, then $A_{1}$ also runs in time $\operatorname{poly}(\frac{n}{\varepsilon})$.

Proof.

We define algorithm $A_{1}^{{\mathcal{S}}(\cdot)}$ on input $x$ as follows: $A_{1}$ first computes $f(x_{1})$ to get $x^{\prime}_{1}$. Then it creates a set ${\mathcal{S}}^{\prime}=\left\{x\in{\mathcal{X}}_{2}\colon\Pr[g(x)\in{\mathcal{S}}]\geq 1/2\right\}$ and runs $A_{2}^{{\mathcal{S}}^{\prime}(\cdot)}$ on $x^{\prime}_{1}$ to get $x^{\prime}_{2}$. Then, it computes $g(x^{\prime}_{2})$ for at most $n$ times until it gets some $x_{2}\in{\mathcal{S}}$ and outputs $x_{2}$, otherwise it outputs a fresh $g(x^{\prime}_{2})$. We have

[TABLE]

Note that the oracle ${\mathcal{S}}^{\prime}(\cdot)$ cannot be implemented in polynomial time, but it could be approximated with negligible error in polynomial time. On the other hand, we have

[TABLE]

∎

The following construction shows an embedding from Gaussian distribution to hamming cube. Using this embedding and Lemma 4.2 we get computational concentration for the Gaussian distribution. The following embedding uses ideas similar to [B*+*97].

Construction 4.3** (CC reduction from (Gaussian, $\ell_{1}$) to Hamming cube).**

We construct $f$ and $g$ as follows.

• $f$

: Let $n$ be an even number. Given a point $x=(x_{1},\dots,x_{n})$ sampled from Gaussian space of dimension $n$, do the following:

If $\exists i;|x_{i}|\geq\sqrt{n}/2$, output $0^{n^{2}}$. 2. 2.

Otherwise, for each $i\in n$ compute $a_{i}=[\frac{x_{i}}{\sqrt{n}}+\frac{n}{2}]$ then uniformly sample some $y_{i}\in\left\{0,1\right\}^{n}$ such that $y_{i}$ has exactly $a_{i}$ number of $1$s. Then append $y_{i}$ s to get $y=(y_{1}|\dots|y_{n})$.

• $g$

: Let $y=(y_{1}|\dots|y_{n})$ be a Boolean vector of size $n^{2}$ (each $y_{i}$ has size $n$). Let $a_{i}$ be the number of $1$s in $y_{i}$. Then sample $x=(x_{1},\dots,x_{n})$ from Gaussian space conditioned on $\frac{2a_{i}-n}{2\sqrt{n}}\leq x_{i}<\frac{2a_{i}-n+1}{2\sqrt{n}}$

Claim 4.4**.**

The embedding of Construction 4.3 is an $(\operatorname{negl}(n),1/\sqrt{n},1/\sqrt{n})$ CC reduction from Gaussian space under $\ell_{1}$ to Hamming cube (i.e., Boolean hypercube under Hamming distance).

Proof.

The embedding property of these mappings is proved in [B*+*97]. The mappings $f$ and $g$ are clearly polynomial time in $n$ and the Almost Lipschitz and Inverse Mappings properties are straightforward. ∎

The following Corollary follows from Lemma 4.2, Claim 4.4 and Theorem 3.2.

Corollary 4.5** (Computational concentration of Gaussian under $\ell_{1}$).**

There is an algorithm $A_{\varepsilon,\delta}^{{\mathcal{S}},{\bm{\upmu}}}(\cdot)$ that given access to a membership oracle for any set ${\mathcal{S}}$ and a sampling oracle from an isotropic Gaussian measure ${\bm{\upmu}}$ of dimension $n$, it achieves the following. If $\Pr[{\mathcal{S}}]\geq\varepsilon$, given $\varepsilon$ and $\delta$, the algorithm $A_{\varepsilon,\delta}^{{\mathcal{S}},{\bm{\upmu}}}(\cdot)$ runs in time $\operatorname{poly}(\nicefrac{{n}}{{\varepsilon\delta}})$, and with probability $\geq 1-\delta$ given a random point $\overline{x}\leftarrow{\bm{\upmu}}$, it maps $\overline{x}$ to a point $\overline{y}\in{\mathcal{S}}$ of bounded $\ell_{1}$ distance $\ell_{1}(\overline{x},\overline{y})\leq O(\sqrt{n\cdot\ln(\nicefrac{{1}}{{\varepsilon\delta}})})$.

4.1 Case of Gaussian or Sphere under $\ell_{2}$

A reduction may also be used to obtain a (non-optimal) computational concentration of measure for the multi-dimensional Gaussian distribution under the $\ell_{2}$ metric.

Theorem 4.6**.**

There is an algorithm $A_{\varepsilon,\delta}^{{\mathcal{S}},{\bm{\upmu}}}(\cdot)$ that given access to a membership oracle for any set ${\mathcal{S}}$ and a sampling oracle from an isotropic Gaussian measure ${\bm{\upmu}}$ of dimension $n$, where each coordinate has variance 1, it achieves the following. If $\Pr[{\mathcal{S}}]\geq\varepsilon$, given $\varepsilon,\delta\geq 1/n^{O(1)}$, the algorithm $A_{\varepsilon,\delta}^{{\mathcal{S}},{\bm{\upmu}}}(\cdot)$ runs in time $\operatorname{poly}(n)$, and with probability $\geq 1-\delta$ given a random point $\overline{x}\leftarrow{\bm{\upmu}}$, it maps $\overline{x}$ to a point $\overline{y}\in{\mathcal{S}}$ of bounded $\ell_{2}$ distance $\ell_{2}(\overline{x},\overline{y})\leq O(n^{1/4}\log^{O(1)}n)$.

Proof.

Since $\epsilon\geq 1/n^{O(1)}$, at most $\epsilon/2$ and $\delta/2$ fraction of the points have a coordinate of size $\geq O(\sqrt{\log n})$. So ignoring points having such large coordinates, we may assume $\Pr[{\mathcal{S}}]\geq\epsilon/2$ while every point of ${\mathcal{S}}$ has coordinates as small as $O(\sqrt{\log n})$, and we may assume the point we are mapping also has small coordinates (except our algorithm should now work for $1-\delta/2$ fraction of the points instead of for $1-\delta$ fraction.)

Now, when each coordinate is $O(\sqrt{\log n})$, the $l_{2}$ distance between two points is at most $O(\sqrt{d_{H}\log n})$, where $d_{H}$ is the Hamming distance of the two points. Now, the theorem follows from our main theorem for Hamming distance. ∎

We should note that the above computational bound is not information-theoretically tight, since for the Gaussian $\ell_{2}$ metric probability space, where each coordinate has variance 1, the right bound is $O(\sqrt{\ln(1/(\epsilon\delta))})$. (This follows e.g. from the Gaussian isoperimetric inequality proved in [ST78, Bor75], which shows the half-space is isopermetrically optimal for the Gaussian distribution.)

Finally, the following shows that our results are not limited to product spaces, and may for example be applied to computational concentration of measure for the high-dimensional sphere.

Theorem 4.7**.**

There is an algorithm $A_{\varepsilon,\delta}^{{\mathcal{S}},{\bm{\upmu}}}(\cdot)$ that given access to a membership oracle for any set ${\mathcal{S}}$ and a sampling oracle from the uniform measure ${\bm{\upmu}}$ on the unit sphere of dimension $n$, it achieves the following. If $\Pr[{\mathcal{S}}]\geq\varepsilon$, given $\varepsilon,\delta\geq 1/n^{O(1)}$, the algorithm $A_{\varepsilon,\delta}^{{\mathcal{S}},{\bm{\upmu}}}(\cdot)$ runs in time $\operatorname{poly}(n)$, and with probability $\geq 1-\delta$ given a random point $\overline{x}\leftarrow{\bm{\upmu}}$, it maps $\overline{x}$ to a point $\overline{y}\in{\mathcal{S}}$ of bounded $\ell_{2}$ distance $\ell_{2}(\overline{x},\overline{y})\leq O(n^{-1/4}\log^{O(1)}n)$.

Proof.

First, we note that a random Gaussian vector, where each coordinate has variance 1, has $\ell_{2}$ norm $\sqrt{n}+O(n^{1/4})$ except for arbitrary inverse polynomial probability.

So given $\overline{x}$, we can map it to a new vector $\overline{x}^{\prime}$ with the same direction as $\overline{x}$ but with a random length of distribution square root of chi square, so that the new vector has the Gaussian distribution. We also map the set ${\mathcal{S}}$ to the set ${\mathcal{S}}^{\prime}=\{r\cdot s:r\in n^{1/2}+O(n^{1/4}),s\in{\mathcal{S}}\}$, where the new set still has probability $\geq\epsilon/2$ under the Gaussian distribution. By the computational concentration of measure for the Gaussian, we know that we can map, with probability $1-\delta/2$, $\overline{x}^{\prime}$ to a point $\overline{y}^{\prime}\in{\mathcal{S}}^{\prime}$ of distance $n^{1/4}\log^{O(1)}n$ from $\overline{x}^{\prime}$ in $\ell_{2}$. Let $\overline{y}$ be the projection of $\overline{y}^{\prime}$ onto the unit sphere. Therefore

[TABLE]

∎

These types of relations between concentration of measure of Gaussian and uniform sphere measures has been well-known information-theoretically, e.g. see [Led01, page 2] where concentration for Gaussian is derived from concentration for sphere. In the above we showed a similar relation for computational concentration of measure, this time deriving for the sphere from the Gaussian.

5 Computational Concentration around Mean

Let $({\mathcal{X}},{{\mathsf{d}}},{\bm{\upmu}})$ be a metric probability space and $f\colon{\mathcal{X}}\mapsto{\mathbb{R}}$ a measurable function (with respect to ${\bm{\upmu}}$). For any Borel set ${\mathcal{T}}\subseteq{\mathbb{R}}$, an parameters $k,\delta\in{\mathbb{R}}_{+}$, one can define a computational problem as follows. Given oracle access to a sampler from ${\bm{\upmu}}$, ${{\mathsf{d}}}$ and function $f(\cdot)$, map a given input $x\in{\mathcal{X}}$ algorithmically to $y\in{\mathcal{Y}}$, such that: (1) ${{\mathsf{d}}}(x,y)\leq k$, and (2) $f(y)\in{\mathcal{T}}$ for $1-\delta$ fraction of $x\in{\mathcal{X}}$ according to ${\bm{\upmu}}$. If we already know that $({\mathcal{X}},{{\mathsf{d}}},{\bm{\upmu}})$ is $(\varepsilon,\delta,k)$ (computationally) concentrates, and if $\Pr_{x\leftarrow{\bm{\upmu}}}[f(x)\in{\mathcal{T}}]\geq\varepsilon$, then it implies that by changing $x$ by at most distance $k$ into a new point $y$, we can (algorithmically) get $f(y)\in{\mathcal{T}}$, by defining ${\mathcal{S}}=f^{-1}({\mathcal{T}})$ and noting that $\Pr_{\bm{\upmu}}[{\mathcal{S}}]\geq\varepsilon$. This algorithm needs oracle access to ${\mathcal{S}}$

Computational concentration around mean.

Again, let $({\mathcal{X}},{{\mathsf{d}}},{\bm{\upmu}})$ be a metric probability space and let $f\colon{\mathcal{X}}\mapsto{\mathbb{R}}$ be measurable. Now suppose $\eta=\operatorname*{\mathbb{E}}_{x\leftarrow{\bm{\upmu}}}[f(x)]$. If we already know, by information theoretic concentration bounds, that $\Pr_{x\leftarrow{\bm{\upmu}}}[|f(x)-\eta|\leq T]\geq 1-\delta$, then it means that a trivial algorithm that does not even change given $x\leftarrow{\bm{\upmu}}$, finds a point where $f(x)$ is $T$-close to the average $\eta$. However, this becomes nontrivial, if the goal of the algorithm is to find $y$ that is close to $x$, and that $f(y)$ is much closer to the mean $\eta$ than what $x$ achieves. In particular, suppose we somehow know that $\Pr_{x\leftarrow{\bm{\upmu}}}[|f(x)-\eta|\leq t]\geq\varepsilon$ for $t\ll T,\varepsilon\ll 1-\delta$. (Such results usually follow from the same concentration inequalities proving $\Pr_{x\leftarrow{\bm{\upmu}}}[|f(x)-\eta|\leq T]\approx 1$.) The smaller $t$ is, the “higher quality” the point $x$ has in terms of $f(x)$ being closer to the mean. This means the set ${\mathcal{S}}=\left\{x\colon|f(x)-\eta|\leq t\right\}$ has ${\bm{\upmu}}$ measure at least $\varepsilon$. Therefore, if the space $({\mathcal{X}},{{\mathsf{d}}},{\bm{\upmu}})$ is $(\varepsilon,\delta,k)$ computationally concentrated, then we can conclude that there is an efficient algorithm (whose running time can polynomially depend on $1/\varepsilon\delta$ and) that maps $1-\delta$ fraction of $x\leftarrow{\bm{\upmu}}$ to a point $y\in{\mathcal{S}}$. Different, but similar, statements about one-sided concentration can be made as well, if we start from weaker conditions of the form $\Pr_{x\leftarrow{\bm{\upmu}}}[f(x)>\eta+t]\leq\varepsilon$ (or $\Pr_{x\leftarrow{\bm{\upmu}}}[f(x)<\eta-t]\leq 1-\varepsilon$) leading to a weaker conclusion: we can map $x$ to a point $y$ satisfies $f(x)\geq\eta-t$ (or $f(x)\leq\eta+t$).

Finally, we note that even if the mean $\eta$ is not known to the mapping algorithm $A$, good approximations of it can be obtained by repeated sampling and taking their average. So for simplicity, and without loss of generality, the reader can assume that $\eta$ is known to the mapping algorithm $A$.

Special case of Lipschitz functions: algorithmic proofs of concentration.

When $f\colon{\mathcal{X}}\mapsto{\mathbb{R}}$ is Lipschitz, i.e., $|f(x)-f(y)|\leq{{\mathsf{d}}}(x,y)$, computational concentration around a set like ${\mathcal{S}}=\left\{x\colon|f(x)-\eta|\leq t\right\}$ (or similar one-sided variants) means something stronger than before. We now have an algorithm that indirectly proves the concentration around $\eta$ by efficiently finding points that are almost at the border defined by $\eta$. Namely, the Lipschitz now implies that $|f(x)-f(y)|\leq k$, whenever $|x-y|\leq k$. Therefore, the algorithm $A$ mapping $x$ to $y$ is also proving that $1-\delta$ measure of the space $({\mathcal{X}},{\bm{\upmu}})$ is mapped under $f$ to a point that is $k+t$ close to average $\eta$.

All the above arguments are general and apply to any metric probability space. Below, we discuss an special case of a “McDiarmid type” inequality in more detail to demonstrate the power of this argument.

Theorem 5.1** (An algorithmic variant of McDiarmid inequality).**

Suppose ${\bm{\upmu}}\equiv{\bm{\upmu}}_{1}\times\dots\times{\bm{\upmu}}_{n}$ is a product measure on a product space ${\mathcal{X}}={\mathcal{X}}_{1}\times\dots\times{\mathcal{X}}_{n}$, and let $f\colon{\mathcal{X}}\mapsto{\mathbb{R}}$ be such that $|f(\overline{x})-f(\overline{x}^{\prime})|\leq\alpha_{i}$ whenever $\overline{x}$ and $\overline{x}^{\prime}$ only differ in the $i^{\mathrm{th}}$ coordinate. Let $a=\left\lVert{\overline{\alpha}}\right\rVert_{2}$ for ${\overline{\alpha}}=(\alpha_{1},\dots,\alpha_{n})$. Let $\eta=\operatorname*{\mathbb{E}}_{x\leftarrow{\bm{\upmu}}}[f(\overline{x})]$ and ${\mathcal{S}}=\left\{x\colon f(\overline{x})\leq\eta+\varepsilon\cdot a\right\}$. Then there is an algorithm $A_{\varepsilon,\delta}^{{\bm{\upmu}},f(\overline{x})}(\cdot)$ running in time $\operatorname{poly}(\nicefrac{{n}}{{\varepsilon\delta}})$ that uses oracle access to $f$ and a sampler from ${\bm{\upmu}}$, and it holds that

[TABLE]

Corollaries for special cases.

Theorem 5.1 implies a similar result when the quality of the destination region is base on the $\ell_{1}$ norm; namely, ${\mathcal{S}}=\left\{x\colon f(\overline{x})\leq\eta+\varepsilon\cdot\left\lVert\alpha\right\rVert_{1}\right\}$, but this follows from the same statement since $\left\lVert\alpha\right\rVert_{2}\leq\left\lVert\alpha\right\rVert_{1}$. In addition, for the special case where $\alpha_{i}=1$ for all $i$,888For example, this could be the setting of Hoeffding’s inequality in which each coordinate ${\bm{\upmu}}_{i}$ is arbitrarily distributed over $[0,1]$, and $f(\overline{x})=\sum_{i\in[n]}x_{i}$, where $\overline{x}=(x_{1},\dots,x_{n})$ and let $\gamma,\delta=1/\operatorname{poly}(n)$ be arbitrarily small inverse polynomials. In that case, Theorem 5.1, shows that for $1-\delta$ fraction of $\overline{x}\leftarrow{\bm{\upmu}}$, we can map $\overline{x}$ to $\overline{y}$ in $\operatorname{poly}(n)$ time in such a way that $f(\overline{y})\leq\operatorname*{\mathbb{E}}[f({\bm{\upmu}})]+\gamma$ and $|f(\overline{x})-f(\overline{y})|\leq\widetilde{O}(\sqrt{n})$. If we choose $\gamma<1/2$, due to the Lipschitz condition, we can also find some $\overline{y}$ for which $f(\overline{y})\in\operatorname*{\mathbb{E}}[f({\bm{\upmu}})]\pm 1$. This is possible by first finding some $\overline{y}$ where $f(\overline{y})\leq\operatorname*{\mathbb{E}}[f({\bm{\upmu}})]+\gamma$, and then go back over the coordinates in which $\overline{x}$ and $\overline{y}$ differ and only changing some of them to get $\overline{y}^{\prime}$ where $f(\overline{y}^{\prime})\in\operatorname*{\mathbb{E}}[f({\bm{\upmu}})]\pm 1$, and output $\overline{y}^{\prime}$ instead. We note that, however, that whenever we want to choose $\gamma<1/2$, we need to also choose $\varepsilon<1/(2n)$. For this range of small $\varepsilon$, we cannot use the computational concentration results of [MM19], but we can indeed use the stronger computational concentration results of this work that prove computational concentration around any non-negligible event.

Proof of Theorem 5.1.

For starters, suppose $\eta$ is given. In that case, we first observe that $\Pr_{\bm{\upmu}}[{\mathcal{S}}]\geq 1-e^{-2\varepsilon^{2}}=\Theta(\varepsilon^{2})$ by McDiarmid’s inequality itself. We can then apply Theorem 3.2.

When $\eta$ is not given, we can find a sufficiently good approximation of it, such that $\eta^{\prime}\in\eta\pm\left\lVert\alpha\right\rVert_{2}\cdot\varepsilon/10$ (in time $\operatorname{poly}(\nicefrac{{n}}{{\varepsilon\delta}})$ and error probability $\delta/10$) and use it instead of $\eta$. Obtaining such $\eta^{\prime}$ can be done because any $x,x^{\prime}$ satisfy $|f(x)-f(x^{\prime})|\leq\left\lVert\alpha\right\rVert_{1}$. Therefore, we can obtain $\eta^{\prime}\in\eta\pm\lambda\cdot\left\lVert\alpha\right\rVert_{1}$ in time by sampling $\ell=\operatorname{poly}(\nicefrac{{n}}{{\lambda\delta}})$ (for sufficiently large $\ell$) many points $\overline{x}_{1},\dots,\overline{x}_{\ell}\leftarrow{\bm{\upmu}}$ and letting $\eta^{\prime}=\operatorname*{\mathbb{E}}_{i\leftarrow\ell}f(\overline{x}_{i})$. The only catch is that we want $\eta^{\prime}\in\eta\pm\varepsilon\cdot\left\lVert\alpha\right\rVert_{2}$. However, since it holds that $\left\lVert\alpha\right\rVert_{2}\leq\left\lVert\alpha\right\rVert_{1}\cdot\sqrt{n}$, we can choose $\lambda=\varepsilon/\sqrt{n}$, and use the same procedure to obtain $\eta^{\prime}\in\eta\pm\lambda\cdot\left\lVert\alpha\right\rVert_{1}$ with probability $1-\delta/10$ in time $\operatorname{poly}(\nicefrac{{n}}{{\varepsilon\delta}})$. ∎

6 Limits of Nonadaptive Methods for Proving Computational Concentration

In this section, we consider three restricted types of attacks and prove exponential lower bounds on their running time. The attacks are

• •

I.i.d. queries: An attack where given $\overline{x}$, we query i.i.d. points whose distribution may depend on $\overline{x}$, until one of these points lies in ${\mathcal{S}}$. The analysis of this attack boils down to analysis of a single-query attack where we want to maximize the probability of ${\mathcal{S}}$-membership of the queried point.

• •

Non-adaptive queries: An attack where given $\overline{x}$, we output a list of points, and query all the points in this list. Since the points in the list are determined before the querying, this attack is non-adaptive. It is easy to see (and we give a proof below) how lower bounding this type of attack reduces to the previous type of attack.

• •

Querying only points close enough to have a chance to be output: If we are interested in finding a point at distance $\leq d$ from $\overline{x}$, one may be tempted to limit the queried points to points at distance $\leq d$ from $\overline{x}$. We show how lower bounding this type of attack reduces to the previous type of attack.

Theorem 6.1** (Lower bound for non-adaptive algorithms).**

Let ${\bm{\upmu}}$ be the uniform probability distribution on $\{1,-1\}^{n}$, and let $\varepsilon=1/2$ and $\delta<1/2$ be constants. There does not exist any non-adaptive algorithm $A$ that given $\overline{x}\leftarrow{\bm{\upmu}}$, the algorithm outputs $m=n^{O(1)}$ (random) points $\overline{y}^{1},\ldots,\overline{y}^{m}$, all within Hamming distance $n^{1-\Omega(1)}$ of $\overline{x}$, such that given any set ${\mathcal{S}}$ with $\Pr[{\mathcal{S}}]\geq\varepsilon$, one of these $m$ points lies in ${\mathcal{S}}$ with probability $1-\delta$ over the randomness of $x$ and randomness of $\overline{y}^{1},\ldots,\overline{y}^{m}$.

Proof.

Assume for the sake of contradiction that such an algorithm $A$ exists. Consider the following modified algorithm: given $\overline{x}$, run $A$ to produce $\overline{y}^{1},\ldots,\overline{y}^{m}$, and then let $\overline{z}^{1}$ be one of those $m$ vectors uniformly at random. To produce $\overline{z}^{2}$, run $A$ independently afresh, and let $\overline{z}^{2}$ be one of the $m$ freshly produced vectors. We can continue in this way, and produce the vectors $\overline{z}^{1},\ldots,\overline{z}^{m^{\prime}}$ as the output of the modified algorithm. By the assumption, for any constant $\delta^{\prime}\in(\delta,1/2)$, with probability $1-\delta^{\prime}$ over the randomness of $\overline{x}$, algorithm $A$ has success probability at least $1/n$, hence each $\overline{z}^{i}$ lies in ${\mathcal{S}}$ with probability $\geq 1/mn$. Hence for these $\overline{x}$, if we choose $m^{\prime}=mn^{2}$, with probability $1-(1-1/mn)^{m^{\prime}}=1-o(1)$, the modified algorithm succeeds. Therefore, the average success probability of the algorithm is $\geq 1-\delta^{\prime}-o(1)\geq 1/2+\Omega(1)$.

The above argument shows that we only need to look at algorithms where $\overline{y}^{1},\ldots,\overline{y}^{m}$ are independent given $\overline{x}$. Thus, it is enough to show that there does not exist a random mapping from $\overline{x}$ to a vector $\overline{y}$ in such a way that with probability $1-\delta$ over the randomness of $\overline{x}$, the probability $\Pr[\overline{y}\in{\mathcal{S}}]$ is non-negligible (since $m$ is polynomial in $n$).

For the sake of contradiction, assume such a mapping from $\overline{x}$ to $\overline{y}$ exists. Let ${\mathcal{S}}$ be a random half-space, i.e. ${\mathcal{S}}=\{\overline{z}:\sum_{i=1}^{n}a_{i}z_{i}\leq 0\}$ for a uniformly random vector $a=(a_{1},\ldots,a_{n})\in\{-1,1\}^{n}$. We will show that for every $\overline{x}$, with probability $\delta$ over the randomness of $a$, the probability $\Pr[\overline{y}\in{\mathcal{S}}]$ is negligible. By an averaging argument, this shows that there exists a half-space ${\mathcal{S}}$ such that with probability $\delta$ over the randomness of $\overline{x}$, $\Pr[\overline{y}\in{\mathcal{S}}]$ is negligible, completing the proof.

As mentioned above, we want to show that for every $\overline{x}$, a random half-space is troublesome for the algorithm. By symmetry, without loss of generality, we may assume $\overline{x}=(1,1,\ldots,1)$. Let $\eta=(\eta_{1},\ldots,\eta_{n})=(\overline{x}-\overline{y})/2$ be the characteristic vector for the coordinates for which $\overline{y}$ is different from $\overline{x}$. We note that $\overline{y}\in{\mathcal{S}}$ iff $\sum_{i}a_{i}-2\sum_{i}a_{i}\eta_{i}\leq 0$. We know that with probability $\delta+\Omega(1)$ over the randomness of $a$, we have $\sum_{i}a_{i}\geq\Omega(\sqrt{n})$. (This easily follows from the central limit theorem.) Now, conditioned on $\eta$, the sum $\sum_{i}\eta_{i}a_{i}$ is actually a sum of $n^{1-\Omega(1)}$-many $\pm 1$ independent random variables of mean zero, so $\Pr[\sum_{i}\eta_{i}a_{i}\geq\Omega(\sqrt{n})]$ is a negligible, actually exponentially small, probability. This implies over the randomness of $a$ and $\eta$, $\Pr[\sum_{i}\eta_{i}a_{i}\geq\Omega(\sqrt{n})]$ is negligible. Thus, except for an $o(1)$ fraction of random half-spaces, $\Pr[\sum_{i}\eta_{i}a_{i}\geq\Omega(\sqrt{n})]$ is negligible over the randomness of $\overline{y}$. Thus, with probability at least $\delta+\Omega(1)-o(1)\geq\delta$ over the randomness of $a$, we have both

• •

$\sum_{i}a_{i}\geq\Omega(\sqrt{n})$, and

• •

$\Pr[\sum_{i}a_{i}\eta_{i}=\Omega(\sqrt{n})]$ is negligible over the randomness of $\overline{y}$.

In this case, $\overline{y}$ does not lie in ${\mathcal{S}}$ except with non-negligible. ∎

Remark 6.2**.**

It can be seen that the above theorem holds whenever $\varepsilon$ and $\delta$ are positive constants such that $\varepsilon+\delta<1$. It can be seen that the above theorem does not hold when $\varepsilon+\delta>1$ since when we set $\overline{y}=\overline{x}$, our failure probability $\delta$ is exactly $1-\varepsilon$.

Lemma 6.3**.**

Given a radius $r$, assume an adaptive algorithm $A$, given $\overline{x}$, wants to find a vector $\overline{y}\in{\mathcal{S}}$ in the ball of radius $r$ around $\overline{x}$. Furthermore, assume that the algorithm does not make any ${\mathcal{S}}$-membership oracle queries regarding points outside the ball. Then, we can transform the algorithm into a non-adaptive algorithm with the same performance.

Proof.

When the algorithm ever queries about a point $\overline{y}$ (and by assumption $\overline{y}$ is in the ball), if the oracle says that $\overline{y}\in S$, then we are done (since we have found our desired point.) So the algorithm may always pretend that the result of each membership query about each queried point is that the point is not in ${\mathcal{S}}$. This equivalent algorithm is non-adaptive. ∎

Corollary 6.4**.**

In the $\{0,1\}^{n}$ uniform product space, when we want to find a point $\overline{y}\in{\mathcal{S}}$ at distance $n^{1/2+\varepsilon}$ from a random $\overline{x}$ (for some $\varepsilon\in(0,1/2)$), to be query-efficient, we need to query about ${\mathcal{S}}$-membership of points having distance more than $n^{1/2+\varepsilon}$.

The above corollary says that even though we are interested in points in a ball of certain radius around $\overline{x}$, we have to query about points outside that ball. When we notice that we are not assuming any structure on the set ${\mathcal{S}}$ other than it should have some minimum mass, the above corollary becomes all the more surprising!

7 Acknowledgement

We would like to thank Amin Aminzadeh Gohari, Salman Beigi, and Mohammad Hossein Yassaee for helpful discussions.