Learning from satisfying assignments under continuous distributions

TL;DR

This paper explores the learnability of distributions over continuous domains defined by the satisfying assignments of low-complexity Boolean functions, providing algorithms and hardness results for different degrees of polynomial threshold functions under various distributions.

Contribution

It extends the framework of learning from satisfying assignments to continuous domains, analyzing the learnability of distributions defined by low-degree polynomial threshold functions under different background distributions.

Findings

Efficient learning algorithms for degree-1 PTFs under log-concave distributions.

Hardness results for degree-2 PTFs under log-concave distributions.

Efficient algorithms for degree-2 PTFs under normal distributions.

Abstract

What kinds of functions are learnable from their satisfying assignments? Motivated by this simple question, we extend the framework of De, Diakonikolas, and Servedio [DDS15], which studied the learnability of probability distributions over $\{0,1\}^n$ defined by the set of satisfying assignments to "low-complexity" Boolean functions, to Boolean-valued functions defined over continuous domains. In our learning scenario there is a known "background distribution" $\mathcal{D}$ over $\mathbb{R}^n$ (such as a known normal distribution or a known log-concave distribution) and the learner is given i.i.d. samples drawn from a target distribution $\mathcal{D}_f$, where $\mathcal{D}_f$ is $\mathcal{D}$ restricted to the satisfying assignments of an unknown low-complexity Boolean-valued function $f$. The problem is to learn an approximation $\mathcal{D}'$ of the target distribution $\mathcal{D}_f$ which has small error as measured in total variation distance. We give a range of efficient algorithms and hardness results for this problem, focusing on the case when $f$ is a low-degree polynomial threshold function (PTF). When the background distribution $\mathcal{D}$ is log-concave, we show that this learning problem is efficiently solvable for degree-1 PTFs (i.e.,~linear threshold functions) but not for degree-2 PTFs. In contrast, when $\mathcal{D}$ is a normal distribution, we show that this learning problem is efficiently solvable for degree-2 PTFs but not for degree-4 PTFs. Our hardness results rely on standard assumptions about secure signature schemes.

Equations

$$\sum _{i=1}^n{\lambda }_i(x_i-{\mu }_i{)}^2+c.$$

$$\sum _{i=1}^n{\lambda }_i({\mathbf{G}}_i-{\mu }_i{)}^2\ge C.$$

$$\int \frac{1}{2}\tilde{\mu }(x)\le \int {\tilde{\mu }}_T(x)\le \int \tilde{\mu }(x)dx.$$

$$\frac{1}{1+\epsilon }{\mathrm{Pr}}_{\mathbf{x}\leftarrow \mathcal{D}}\left[f(\mathbf{x})=1\right]\le \hat{p}\le (1+\epsilon ){\mathrm{Pr}}_{\mathbf{x}\leftarrow \mathcal{D}}\left[f(\mathbf{x})=1\right].$$

$$\frac{1}{1+\epsilon }{\mathcal{D}}_f(x)\le {\hat{\mathcal{D}}}_{f,\epsilon }(x)\le (1+\epsilon ){\mathcal{D}}_f(x)$$

$$d_{\mathrm{TV}}({\mathcal{D}}_f,{\mathcal{D}}_{f,\tau }^{\prime })\le \tau$$

$$(1-\epsilon )\cdot {\mathrm{Pr}}_{\mathbf{x}\leftarrow \mu }[g(\mathbf{x})=1]\le \hat{p}\le (1+\epsilon )\cdot {\mathrm{Pr}}_{\mathbf{x}\leftarrow \mu }[g(\mathbf{x})=1].$$

$$1-\epsilon \le \frac{Q}{{\int }_x\tilde{\mu }(x)dx}\le 1+\epsilon .$$

$${\mu }_{T,z}(x)=\frac{1}{\det (T)}\mu (T^{-1}(x-z)).$$

$${\int }_x\tilde{\mu }(x)dx={\int }_x{\tilde{\mu }}_{T,z}(x)dx$$

$$\frac{\sigma (x)}{{\mu }_g(x)}\le 2^{2n^C+9n+3}\cdot n^{2Cn}.$$

$$\frac{{\sigma }_{T,z}(x)}{({\mu }_{T,z}{)}_g(x)}\le 2^{2n^C+9n+3}\cdot n^{2Cn}.$$

$$\frac{1}{\beta }\le {\int }_{x\in {\mathbb{R}}^n}⟨u,x{⟩}^2\mu (x)dx\le \beta .$$

$$f(z)\leq\min_{x\in K}f(x)+\varepsilon\cdot\big{(}\max_{x\in K}f(x)-\min_{x\in K}f(x)\big{)}$$

$$2^{-2n^C-7n-3}\cdot n^{-2Cn}\le {\mu }_g(x)\le 2^{2n}$$

$$\underset{x\in K}{\max }h(x)-\underset{x\in K}{\min }h(x)\le 2n^C+2Cn\log n+9n+3.$$

$$R^{\prime }=2n^C,r^{\prime }=2\cdot (2n{)}^{-(n-1)C}\cdot {\pi }^{-(n-1)}\cdot 2^{-3n},$$

$$\mathsf{surf}(K)=\underset{\delta \to 0}{\lim }\frac{\mathsf{vol}(K_{\delta })-\mathsf{vol}(K)}{\delta }.$$

$$\mathsf{vol}(K_t)-\mathsf{vol}(K)={\int }_{s=0}^t\mathsf{surf}(K_s)ds.$$

$$\mathsf{vol}(A_{g,r^{\prime }})-\mathsf{vol}(A_g)\le {\int }_{s=0}^{r^{\prime }}2{\pi }^{n-1}\cdot (n^C+s{)}^{n-1}ds$$

$$\epsilon :=\frac{2^{-2n-1}}{2n^C+2Cn\log n+9n+3}.$$

$$h(z)\leq\min_{x\in K}h(z)+\varepsilon\cdot\big{(}\max_{x\in K}h(x)-\min_{x\in K}h(x)\big{)}\leq\min_{x\in K}h(x)+2^{-2n-1}\ \textrm{ using~{}\eqref{eq:diff-log}.}$$

$${\mathrm{Pr}}_{\mathbf{x}\leftarrow {\mathcal{D}}_f}[\mathbf{x}\in K_g]={\mathrm{Pr}}_{\mathbf{x}\leftarrow {\mathcal{D}}_f}[\mathbf{x}\in g^{-1}(-1)]\ge \epsilon .$$

$$n^{C(n-1)}\cdot {\pi }^{n-1}\cdot 2^n\cdot \kappa \cdot 4^n\ll \frac{\epsilon }{2}$$

$$n^{C(n-1)}\cdot {\pi }^{n-1}\cdot 2^n\cdot \kappa \cdot 4^n\ll 2^{-n}.$$

$$\left|p(x)-p([x{]}_{\kappa })\right|\le O(1)\cdot \kappa \cdot {\Vert x\Vert }_2=O(\kappa n^C):=\tau .$$

$${\mathbf{Pr}\text{/}}_{\mathbf{x}\leftarrow \mathcal{N}(0,1{)}^n}\left[\left|p(\mathbf{x})-\theta \right|<\epsilon \sqrt{\mathrm{Var}[p]}\right]\le O(d{\epsilon }^{1/d}).$$

$${\mathbf{Pr}\text{/}}_{\mathbf{x}\leftarrow {\mathcal{N}}_f}[\mathbf{x}\in K_g\cap [\mathbf{x}{]}_{\kappa }\notin K_g]\le {\mathbf{Pr}\text{/}}_{\mathbf{x}\leftarrow {\mathcal{N}}_f}[\left|p(\mathbf{x})\right|\le \tau ]\le 2^n{\mathbf{Pr}\text{/}}_{\mathbf{x}\leftarrow \mathcal{N}}[\left|p(\mathbf{x})\right|\le \tau ]\le O(2^n\sqrt{\tau })\ll 2^{-n}$$

$$\frac{1}{1+\epsilon }\mathrm{Pr}\left[\sum _{i=1}^n{\mathbf{X}}_i\le C\right]\le \hat{\rho }\le (1+\epsilon )\mathrm{Pr}\left[\sum _{i=1}^n{\mathbf{X}}_i\le C\right].$$

$${\mathbf{Pr}\text{/}}_{\mathbf{G}\sim \mathcal{N}}[p(\mathbf{G})\ge 0]=\mathrm{Pr}\left[\sum _{i=1}^n{\lambda }_i{\mathbf{G}}_i^2+{\mu }_i{\mathbf{G}}_i\le \theta \right].$$

Full text

We give a range of efficient algorithms and hardness results for this problem, focusing on the case when $f$ is a low-degree polynomial threshold function (PTF). When the background distribution $\mathcal{D}$ is log-concave, we show that this learning problem is efficiently solvable for degree-1 PTFs (i.e., linear threshold functions) but not for degree-2 PTFs. In contrast, when $\mathcal{D}$ is a normal distribution, we show that this learning problem is efficiently solvable for degree-2 PTFs but not for degree-4 PTFs. Our hardness results rely on standard assumptions about secure signature schemes.

1 Introduction

Over the past decade or so there has been an explosion of research effort aimed at establishing rigorous results about the learnability of various classes of probability distributions. A natural goal in this line of work is to understand how the complexity of learning an unknown high-dimensional distribution scales with the complexity of the distribution being learned. This immediately leads to a question of how to measure the computational complexity of a high-dimensional distribution; needless to say, a range of different approaches are possible here. In early work, Kearns et al. [KMR*+*94] proposed a model in which the complexity of a distribution $\mathcal{D}$ over $\{0,1\}^{n}$ is measured in terms of the circuit complexity of an $m$-input, $n$-output Boolean circuit $C\colon\{0,1\}^{m}\to\{0,1\}^{n}$ which is such that $\mathcal{D}$ is the output distribution when a uniform random string from $\{0,1\}^{m}$ is given as input to $C$ (i.e., $\mathcal{D}=C(\mathcal{U})$). This is a natural approach, but as Kearns et al. showed, even depth-1 circuits of bounded fan-in OR gates lead to distributions $\mathcal{D}$ which are hard to learn in this model. Moreover, it is not clear how to extend the [KMR*+*94] framework to distributions over the continuous domain $\mathbb{R}^{n}$ as opposed to the Boolean cube $\{0,1\}^{n}.$

More recently [DDS15a] took a different approach by considering “low-complexity” distributions $\mathcal{U}_{f}$ over $\{0,1\}^{n}$ which are uniform over the set of satisfying assignments of a “low-complexity” Boolean function $f\colon\{0,1\}^{n}\to\{-1,1\}.$ They gave a range of algorithms and hardness results for learning distributions $\mathcal{U}_{f}$ of this sort, where the unknown low-complexity Boolean function is assumed to be a linear threshold function, a degree-2 polynomial threshold function, a DNF formula, a monotone width-2 CNF, and so on. The main positive results of [DDS15a] showed that if $f$ is a linear threshold function or a DNF formula, then the distribution $\mathcal{U}_{f}$ is learnable to total variation distance $\varepsilon$ by an efficient algorithm (in time $\mathrm{poly}(n,1/\varepsilon)$ for $f$ a linear threshold function, and in time $n^{O(\log(n/\varepsilon))}$ for $f$ a $\mathrm{poly}(n)$-term DNF formula). [DDS15a] also established a range of hardness results showing that these positive results are close to the limit of what can be achieved by efficient algorithms in their model. They showed that under known constructions of secure cryptographic signature schemes, degree-2 polynomial threshold functions over $\{0,1\}^{n}$ already give rise to distributions which cannot be learned in sub-exponential time. (The same is true for distributions arising from satisfying assignments of monotone width-2 CNF formulas.)

This work: Learning continuous distributions from satisfying assignments

One attractive feature of the [DDS15a] framework is that it can be straightforwardly extended to learning low-complexity distributions over continuous domains. Given a known “background distribution” $\mathcal{D}$ (analogous to the uniform distribution over $\{0,1\}^{n}$) such as a normal distribution or a log-concave distribution over $\mathbb{R}^{n}$, one can naturally consider the problem of learning the distribution $\mathcal{D}_{f}$ induced by restricting $\mathcal{D}$ to the satisfying assignments of an unknown low-complexity Boolean-valued function $f\colon\mathbb{R}^{n}\to\{0,1\}$. It is particularly natural in this setting to take $f$ to be a low-degree polynomial threshold function (PTF), as such functions (unlike, say, DNF or CNF formulas) can very naturally be viewed as functions over the entire domain $\mathbb{R}^{n}$. (Recall that a degree-$d$ PTF over $\mathbb{R}^{n}$ is a function $f\colon\mathbb{R}^{n}\to\{-1,1\}$ defined by $f(x)=\mathrm{sign}(p(x))$ where $p(x_{1},\dots,x_{n})$ is a degree-$d$ real polynomial.)

With this motivation, and the additional motivating hope of establishing efficient learnability of distributions of satisfying assignments over $\mathbb{R}^{n}$ whose discrete analogues over $\{0,1\}^{n}$ do not have efficient learning algorithms, in this paper we consider the aforementioned extension of the [DDS15a] framework to learning continuous distributions of satisfying assignments. We define our learning framework, and describe our results in this framework, below.

1.1 The learning framework

We now introduce our learning model in more detail but still at an informal level; see Section 2 for a detailed description of the model. Similar to [DDS15a], a learning problem in our framework is defined by a class $\mathcal{C}$ of Boolean-valued functions from $\mathbb{R}^{n}$ to $\{-1,1\}$. In our setting an instance of the learning problem is defined by a choice of a function $f\in\mathcal{C}$, which is unknown to the learning algorithm, and a background distribution $\mathcal{D}$, which is provided to the learning algorithm. (We explain in detail just what this means in Section 2; here we note that the distribution $\mathcal{D}$ is analogous to the uniform distribution over $\{0,1\}^{n}$, which is the background distribution in [DDS15a] and is certainly “known” to any learning algorithm.) The learning algorithm has access to i.i.d. samples drawn from $\mathcal{D}_{f}$, which is the distribution $\mathcal{D}$ restricted to the set of satisfying assignments $\left\{\;x\in\mathbb{R}^{n}\;\colon\;f(x)=1\;\right\}$ of $f$. The learner’s task is to output a hypothesis distribution $\mathcal{D}^{\prime}$ (we will amplify on precisely what this means in the next section) which is such that the total variation distance $d_{\mathrm{TV}}(\mathcal{D}_{f},\mathcal{D}^{\prime})$ is at most $\varepsilon$; a successful learning algorithm is one which does this, with high probability over its random samples and any internal randomness, for any $f\in\mathcal{C}$. We say that such an algorithm is a distribution learning algorithm for class $\mathcal{C}$ with respect to background distribution $\mathcal{D}$.

1.2 Our results

The above framework is quite general; the results of [DDS15a] fall into this framework by taking the background distribution ${\cal D}$ to be the uniform distribution over $\{0,1\}^{n}$. It is not difficult to see that if the distribution ${\cal D}$ is totally unrestricted, then learning is intractable in this framework even for simple classes of Boolean functions such as linear threshold functions(see Appendix A). Thus, in this paper we restrict our attention to two different types of background distributions $\mathcal{D}$: (i) log-concave distributions over $\mathbb{R}^{n}$, and (ii) normal distributions over $\mathbb{R}^{n}$. (We recall that a distribution $\mathcal{D}$ over $\mathbb{R}^{n}$ is log-concave if its density function $p$ satisfies $p=\exp(\phi)$ where $\phi\colon\mathbb{R}\to[-\infty,\infty)$ is a concave function.) We further restrict our attention to the function class $\mathcal{C}=\mathsf{PTF}^{n}_{d}$, the class of polynomial threshold functions over $\mathbb{R}^{n}$ of degree at most $d$, where $d$ is some fixed small constant.

Log-concave distributions. Our first positive result is an efficient distribution learning algorithm for linear threshold functions (i.e., PTFs of degree 1) with respect to any log-concave background distribution:

Theorem 1** (Informal statement; see Theorem 5 in Section 2 for detailed statement).**

There is an algorithm $A$, running in time $\mathrm{poly}(n,1/\varepsilon)$, that is a distribution learning algorithm for the class $\mathsf{LTF}$ of $n$-variable linear threshold functions with respect to any log-concave background distribution $\mathcal{D}$ over $\mathbb{R}^{n}$.

We complement this positive result with a negative result for degree-2 PTFs, showing that the above is essentially best possible for log-concave background distributions:

Theorem 2**.**

Under known constructions of secure signature schemes, there is no subexponential-time algorithm $A$ for learning the class $\mathsf{PTF}^{n}_{2}$ of $n$-variable degree-two polynomial threshold functions with respect to every log-concave background distribution $\mathcal{D}$.

Normal distributions. Turning next to normal distributions, our first positive result is an efficient distribution learning algorithm for degree-two polynomial threshold functions with respect to any normal distribution:

Theorem 3** (Informal statement; see Theorem 6 in Section 2 for detailed statement).**

There is an algorithm $A$, running in time $\mathrm{poly}(n,1/\varepsilon)$, that is a distribution learning algorithm for the class $\mathsf{PTF}^{n}_{2}$ of $n$-variable degree-two polynomial threshold functions with respect to any normal background distribution $\mathcal{D}$ over $\mathbb{R}^{n}$.

We complement this positive result with a negative result for degree-4 PTFs:

Theorem 4**.**

Under known constructions of secure signature schemes, there is no subexponential-time algorithm $A$ for learning the class $\mathsf{PTF}^{n}_{4}$ of $n$-variable degree-four polynomial threshold functions with respect to the standard normal distribution ${\cal N}(0,1)^{n}$.

Discussion. Theorems 1 and 2 align closely with the positive and negative results established in [DDS15a] in the discrete setting where the background distribution $\mathcal{D}$ is simply the uniform distribution $\mathcal{U}$ over $\{0,1\}^{n}$. Recall that in [DDS15a] it is shown that there is an efficient distribution learning algorithm for the class of linear threshold functions with respect to the background distribution $\mathcal{U}$, but there is no such efficient distribution learning algorithm for degree-2 PTFs with respect to the background distribution $\mathcal{U}$. In contrast, Theorem 3 shows that normal distributions are “easier” background distributions than uniform or arbitrary log-concave distributions, as satisfying assignments of degree-2 PTFs are efficiently learnable for normal background distributions. But there is a limit to the power afforded by normal background distributions, as Theorem 4 shows that already degree-4 PTFs are hard to learn under normal background distributions.

1.3 Related work

A recent line of work which focuses on learning high-dimensional normal distributions from truncated samples [DGTZ18, KTZ19] has a somewhat similar flavor to ours: in this setting, the underlying distribution is an unknown multivariate normal $\mathcal{D}=\mathcal{N}(\mu,\Sigma)$, and observed samples come from $\mathcal{D}_{f}$, where $f$ is the indicator function of an arbitrary truncation set $S\subseteq\mathbb{R}^{n}$. The goal is to learn the parameters $\mu$ and $\Sigma$ of the true distribution (or, equivalently, to learn $\mathcal{D}$). Daskalakis, Gouleakis, Tzamos, and Zampetakis [DGTZ18] gave an efficient algorithm for this task when the truncation set $S$ is known. An incomparable result of Kontonis, Tzamos, and Zampetakis [KTZ19] gives an efficient algorithm to recover $\mu$ and $\Sigma$ even when $S$ is unknown; however, [KTZ19] require $S$ to have bounded surface area (see [KOS08] for background on this notion) and, more importantly, the covariance matrix $\Sigma$ is required to be diagonal.

In contrast, in our setting the parameters $\mu$ and $\Sigma$ are known to the algorithm; the function $f$ is assumed to be an unknown LTF or degree-$2$ PTF (for our positive results); and the goal is to learn the truncated distribution ${\cal D}_{f}$. One important difference between our results and those of [DGTZ18, KTZ19] is that our algorithm is efficient even when the truncation set $S$ (i.e., the set of satisfying assignments of the unknown LTF or PTF) has inverse exponential density in the Gaussian space; indeed, achieving this is the most challenging aspect of our result. In contrast, in the setting of [DGTZ18, KTZ19], even achieving a sample-efficient algorithm requires that the truncation set $S$ has at least an inverse $\mathrm{poly}(n)$ density. Further, for a computationally efficient algorithm, [KTZ19] requires that the truncation set has a constant density in the Gaussian space.

Finally, another work which is related to the current paper is that of [AGR13], which gives an efficient algorithm for learning an $n$-dimensional simplex when the algorithm is given uniformly random samples from (the uniform distribution on) the simplex. Recall that a simplex in $\mathbb{R}^{n}$ is the same as the intersection of exactly $n+1$ halfspaces along with the requirement that the intersection is a bounded set. Their result and tools seem quite different from ours (as their techniques crucially rely on a reduction to ICA).

1.4 Our upper bound techniques

At the highest level our approach follows the general method for learning from satisfying assignments that was given in [DDS15a], though as described later there are many technical issues and obstacles that are specific to this work. [DDS15a] developed a general framework in which, very roughly, if one is given a number of different types of algorithms for a class $\mathcal{C}$ of functions from $\{0,1\}^{n}$ to $\{0,1\}$, then one obtains a distribution learning algorithm for $\mathcal{C}$ with respect to the background distribution $\mathcal{U}$, the uniform distribution over the Boolean hypercube $\{0,1\}^{n}$. The required algorithms are specifically (i) an approximate uniform generation algorithm for $\mathcal{C}$, (ii) an approximate counting algorithm for $\mathcal{C}$, (iii) a Statistical Query learning algorithm for $\mathcal{C}$, and (iv) a “densifier” for $\mathcal{C}$, explained below. (See Theorem 3.1 of [DDS15a] for a precise statement of the original framework.) We note that approximate uniform generation, approximate counting, and statistical query learning are of course well-studied and standard notions, but the notion of a “densifier” was new to [DDS15a] and a significant amount of the technical work in that paper came in developing densifiers for linear threshold functions and DNF formulas.

As our starting point, we observe that the algorithmic framework of [DDS15a] can be straightforwardly extended to non-uniform background distributions $\mathcal{D}$ by making some natural modifications. In particular,

In place of an approximate uniform generation algorithm (which outputs approximately uniform satisfying assignments of a function $f\in\mathcal{C}$) one needs the natural analogue for distribution $\mathcal{D}$, namely a procedure for approximately sampling from $\mathcal{D}_{f}$. 2. 2.

In place of an approximate counting algorithm for $\mathcal{C}$ (which outputs an approximation of $\left\lvert f^{-1}(1)\right\rvert/2^{n}$, where $f\colon\{0,1\}^{n}\to\{0,1\}$ belongs to $\mathcal{C}$), one needs a procedure for approximate integration of $\mathcal{D}$ over the domain $f^{-1}(1)$. 3. 3.

We also need a densifier with respect to $\mathcal{D}$, which is a natural extension of the densifier notion from [DDS15a]. Roughly speaking, a densifier is an algorithm which is given random positive examples of $f$ (drawn according to ${\cal D}$ restricted to $f^{-1}(1)$) and prunes the entire domain $\mathbb{R}^{n}$ down to a set $S$ which (essentially) contains all of $f^{-1}(1)$ and which is such that $f^{-1}(1)$ is “dense” in $S$ under distribution ${\cal D}$.

(The fourth ingredient, a Statistical Query algorithm, is unchanged from [DDS15a] to the present work, since even the earlier work needed Statistical Query learning with respect to various non-uniform distributions.) In Section 3 we give a precise statement of the extension of [DDS15a]’s Theorem 3.1, Theorem 12, which we will use to obtain our results. In the rest of this subsection we give an overview of how each of the above three ingredients – approximate sampling, approximate integration, and densification, all with respect to $\mathcal{D}$ – are achieved for each of our two positive results (learning linear threshold functions with respect to arbitrary log-concave distributions, and learning degree-2 PTFs with respect to the normal distribution). We begin with the simpler scenario of linear threshold functions.

Learning LTFs with respect to log-concave distributions. The key insight here is that if $\mathcal{D}$ is a log-concave distribution and $f$ is any LTF, then since $f^{-1}(1)$ is a convex set in $\mathbb{R}^{n}$, the distribution $\mathcal{D}_{f}$ is again a log-concave distribution. Our approach leverages this simple observation using known results, due to [LV06], for approximate sampling and approximate integration of log-concave distributions. (We note that significant work is required to align these known results with our setting; see Section 3.1, where Lemma 3.1 formally establishes the approximate sampling and approximate integration results that we require.)

For the densifier, at a high level our approach is similar to the construction of a densifier for linear threshold functions in [DDS15a] where the background distribution is uniform over $\{0,1\}^{n}$. Like [DDS15a], we show how an efficient online mistake-bound algorithm for linear threshold functions can be leveraged to obtain the desired densifier, now with respect to the log-concave background distribution $\mathcal{D}$; however, subtleties arise in our current setting which were not present in [DDS15a] since we are now working in a continuous rather than discrete setting. In particular, online algorithms for learning LTFs with a finite mistake bound do not exist over continuous domains. To circumvent this issue, we use anticoncentration properties of log-concave distributions and the fact that the boundary of $f^{-1}(1)$ is a very “well-structured” set (it is simply a hyperplane in $\mathbb{R}^{n}$) to give an analysis in Section 3.2 showing that, after suitable discretization of the domain, one can apply the discrete-setting mistake-bound algorithms without compromising the required guarantees. Intuitively, this is because a suitably fine discretization only “flips the label” of a tiny fraction of points.

Learning degree-2 PTFs with respect to normal distributions. Conceptually, the densification step for degree-2 PTFs and normal distributions can be handled in roughly the same way as the densification step for the LTF scenario described above. (There are some differences in the technical arguments which show that online mistake-bound learning algorithms can be successfully used in the discretized version of the degree-2 PTF setting. In this setting the boundary of $f^{-1}(1)$ is not simply a hyperplane, so now we must rely on results of Carbery and Wright [CW01] which establish anticoncentration of quadratic polynomials in Gaussian random variables; details are in Section 4.1.) However, new challenges arise in obtaining the required approximate sampling and approximate integration algorithms. The central obstacle is that for $f$ a degree-2 PTF over $\mathbb{R}^{n}$, the set $f^{-1}(1)$ can be non-convex; this means that the resulting restricted normal distribution $\mathcal{N}_{f}$ need not be log-concave, so the known tools for working with log-concave distributions, such as [LV06], are no longer at our disposal. We circumvent this obstacle through a careful approach which heavily exploits the degree-2 polynomial structure of $f$.444This should not be a surprise, recalling our negative results for degree-4 polynomial threshold functions.

We first discuss the approximate integration/counting problem. Via a change of basis, we convert a general degree-2 polynomial to a “decoupled” degree-2 multivariate polynomial, which is a sum of $n$ univariate quadratic polynomials over distinct variables, of the form

[TABLE]

This change of basis does not change the underlying geometry of the problem, but it enables us to reframe the counting/integration problem as the problem of estimating the probability that a random Gaussian $\bm{G}=(\bm{G}_{1},\dots,\bm{G}_{n})$ satisfies a linear inequality

[TABLE]

Crucially, the above polynomial is a sum of independent random variables $\lambda_{i}(\bm{G}_{i}-\mu_{i})^{2}$. We solve this problem by adapting a FPTAS, due to Li and Shi [LS14], for multiplicatively approximating the probability that a sum of independent random variables exceeds a threshold value. The results of [LS14] are for discrete random variables, whereas our setting is continuous; in our adaptation we use anticoncentration results due to Carbery and Wright [CW01] in order to establish that the rounding we perform (so that the [LS14] results can be used) only incurs an acceptably low amount of error. Details are given in Section 4.2.

It remains only to give an efficient routine for approximate sampling from the normal distribution subject to a degree-2 PTF constraint. This is done in Section 4.3, where we extend the usual reduction from approximate counting to approximate sampling and leverage our approximate counting result described in the previous paragraph; an extension is required because the usual reduction is for counting and sampling problems over the domain $\{0,1\}^{n}$, whereas now the relevant domain is $\mathbb{R}^{n}$. We show that it suffices to discretize $\mathbb{R}^{n}$ to an exponentially fine grid, and augment the usual reduction with binary search in each coordinate (which can be carried out efficiently even over an exponential-size domain).

We note that, prior to this work, no (provably) efficient algorithm was known for sampling from a high-dimensional normal subject to quadratic constraints, a task used as a subroutine in many applied settings (see, e.g., [EM07, PP14], and references therein).

1.5 Our lower bound techniques

Our computational hardness results are obtained via an extension of the approach that was given in [DDS15a]. This approach is based on a connection between secure cryptographic signature schemes with certain properties and learning from satisfying assignments. The crux of the idea is very simple when the background distribution is uniform over $\{0,1\}^{n}$ (as in [DDS15a]): if we view $\mathcal{U}_{f}$ as the uniform distribution over signed messages, an algorithm which can construct a sampler for $\mathcal{U}_{f}$ given access to independent draws from it corresponds to an algorithm which, given a batch of signed messages, can generate a new signed message, and this violates the security of a signature scheme. Via reductions, this intuition is leveraged to show that assuming the existence of suitably secure signature schemes, no efficient algorithm can learn various types of functions given access only to random satisfying assignments.

In the current paper we extend this connection to our setting of more general background distributions other than uniform on $\{0,1\}^{n}$. The heart of our lower bound proof for degree-2 PTFs under log-concave distributions is a reduction from Subset-Sum. By designing a suitable degree-2 PTF based on a subset sum instance, we are able to establish hardness even when the background distribution is a very simple and well-structured log-concave distribution, namely the uniform distribution on the solid cube $[0,1]^{n}$; this gives Theorem 2.555Some care is required to ensure that our lower bound construction complies with a natural “reasonableness” requirement we put on learning problems in our framework, which is that the set of all satisfying assignments $f^{-1}(1)$ has at least an inverse exponential amount of probability mass under the background distribution – see the discussion in Section 2 – but this is essentially a technical issue.

Since the background log-concave distribution for our degree-2 PTF hardness result is uniform on $[0,1]^{n}$, which is very “tame,” it is natural to wonder whether this hardness result can be extended to the normal distribution $\mathcal{N}(0,1)^{n}$ as the background distribution. However, this natural intuition is directly contradicted by our main algorithmic result, Theorem 3, which says that degree-2 PTFs are easy to learn under the normal background distribution! Indeed, it turns out that the bounded support of $[0,1]^{n}$ plays an essential role in letting the reduction go through. We show that by working with a suitable degree-four polynomial, it is possible to essentially enforce the bounded support requirement even when the background distribution is $\mathcal{N}(0,1)^{n}$, and we thereby obtain computational hardness for learning degree-four PTFs with respect to the normal background distribution. The learnability of degree-three PTFs with respect to the normal background distribution is an interesting open problem for future work.

2 Preliminaries

Detailed description of our model.

We now explain in detail the sense in which, as stated in Section 1.1, the background distribution $\mathcal{D}$ is “provided to” the learning algorithm.

For the case where $\mathcal{D}$ is a normal distribution, this simply means that the learning algorithm is given, as part of its input, the mean $\mu\in\mathbb{R}^{n}$ and covariance matrix $\Sigma\in\mathbb{R}^{n\times n}$ defining the normal distribution. In fact, since the class of low-degree polynomial threshold functions is closed under affine transformations of the input space $\mathbb{R}^{n}$ (i.e., if $f(x)$ is a PTF of degree $d$, then $f(Ax+b)$ is also a PTF of degree $d$ for every $n\times n$ matrix $A$ and every vector $b\in\mathbb{R}^{n}$), for our normal distribution results we may simply assume that the background distribution is the standard $\mathcal{N}(0,1)^{n}$ distribution. (If the background normal distribution is not full dimensional, then an affine transformation converts it to $\mathcal{N}(0,1)^{k}$ for some $k<n$, and our algorithms can simply be carried out over $\mathbb{R}^{k}$ rather than $\mathbb{R}^{n}.$) Thus we subsequently assume, without loss of generality for our normal distribution results, that the given normal background distribution is simply $\mathcal{N}(0,1)^{n}$.

For the case when $\mathcal{D}$ is a log-concave distribution, the learning algorithm is provided with an evaluation oracle for a function $\tilde{\mu}$ which is proportional to the probability density function $\mu\colon\mathbb{R}^{n}\to[0,1]$ of $\mathcal{D}$. We will assume that $\tilde{\mu}$ satisfies three conditions, where $C>1$ is some sufficiently large absolute constant:

• C1:

all but at most $2^{-Cn}$ of the probability mass is concentrated in the $L_{2}$ ball of radius $n^{C}$;

• C2:

${\lVert\mu{\rVert}}_{\infty}\leq 2^{n}$ and ${\lVert\tilde{\mu}{\rVert}}_{\infty}\leq 2^{n^{C}}$;

• C3:

Let $\tilde{\mu}_{T}(x)$ be obtained by truncating $\tilde{\mu}(x)$ at $n^{C}$ bits after the binary point. Then

[TABLE]

The first two conditions restrict to log-concave distributions that are at least mildly concentrated, and not too “peaked,” while the third condition says that truncating $\tilde{\mu}$ up to polynomial precision provides a good approximation (in a certain weak average sense). We note that these three conditions are quite benign and do not significantly restrict the scope of our results. In particular, they are satisfied by any uniform distribution over a convex set of reasonable size (as well as by the distributions used in our hardness results). We refer to a log-concave distribution that satisfies the above conditions C1 through C3 as being well-behaved.

Next we specify what is exactly meant for the learning algorithm to “output a hypothesis distribution.” What we require is that the learning algorithm output a description of a polynomial-time procedure which has query access to $\tilde{\mu}$ (equivalently, it outputs a polynomial-size circuit with oracle gates for $\tilde{\mu}$) which, on input uniformly random bits, outputs a draw from a distribution $\mathcal{D}^{\prime}$ that is $\varepsilon$-close to $\mathcal{D}_{f}$.

Detailed theorem statements. Before giving detailed theorem statements, we briefly discuss some numerical issues that arise since we are considering distributions over the continuous space $\mathbb{R}^{n}$. If the target function $f\colon\mathbb{R}^{n}\to\{-1,1\}$ and background distribution $\mathcal{D}$ over $\mathbb{R}^{n}$ are such that $f^{-1}(1)$ has extremely low mass under $\mathcal{D}$ — say, $\operatorname{{\bf Pr}}_{\bm{x}\leftarrow\mathcal{D}}[f(\bm{x})=1]=2^{-n^{\omega(1)}}$ — then a super-polynomial number of bits may be required even to differentiate any point in $f^{-1}(1)$ from points in $f^{-1}(-1).$ To avoid such extreme scenarios, we assume for our positive results that the target function $f$ has a “reasonable” (at least inverse exponential) fraction of satisfying assignments under the background distribution. Thus, a detailed statements of our positive results is as follows:

Theorem 5** (Detailed statement of Theorem 1: algorithm for LTFs under log-concave background distribution).**

There is an algorithm $A$ with the following property: Let $\mathcal{D}$ be a well-behaved log-concave distribution provided to $A$ as described above, and let $f$ be a unknown LTF over $\mathbb{R}^{n}$ such that $\operatorname{{\bf Pr}}_{\bm{x}\leftarrow\mathcal{D}}[f(\bm{x})=1]\geq 2^{-n}.$ Given access to i.i.d. samples from $\mathcal{D}_{f}=$ ($\mathcal{D}$ restricted to $f^{-1}(1)$), for $\varepsilon=1/2^{o(n)}$ algorithm $A$ runs in $\mathrm{poly}(n,1/\varepsilon)$ time and with high probability outputs a hypothesis distribution $\mathcal{D}^{\prime}$ such that $d_{\mathrm{TV}}(\mathcal{D}^{\prime},\mathcal{D}_{f})\leq\varepsilon.$

Theorem 6** (Detailed statement of Theorem 3: algorithm for degree-2 PTFs under normal background distribution).**

There is an algorithm $A$ with the following property: Let $f$ be a unknown degree-two PTF over $\mathbb{R}^{n}$ such that $\operatorname{{\bf Pr}}_{\bm{x}\leftarrow{\cal N}(0,1)^{n}}[f(\bm{x})=1]\geq 2^{-n}.$ Given access to i.i.d. samples from $\mathcal{N}_{f}=$ ($\mathcal{N}(0,1)^{n}$ restricted to $f^{-1}(1)$), for $\varepsilon=1/2^{o(n)}$ algorithm $A$ runs in $\mathrm{poly}(n,1/\varepsilon)$ time and with high probability outputs a hypothesis distribution $\mathcal{D}^{\prime}$ such that $d_{\mathrm{TV}}(\mathcal{D}^{\prime},\mathcal{N}_{f})\leq\varepsilon.$

Remark 7**.**

We remark that our hardness results as well as our algorithms will adhere to this “reasonableness” condition: in our constructions establishing computational hardness, the learning problems which we show to be hard are all ones in which the target function $f$ and background distribution $\mathcal{D}$ are such that $\operatorname{{\bf Pr}}_{\bm{x}\leftarrow\mathcal{D}}[f(\bm{x})=1]$ is at least $2^{-\mathrm{poly}(n)}.$

2.1 The framework of [DDS15a] and the ingredients it requires

We now recall the algorithmic components of [DDS15a] which, suitably generalized to the continuous case, will prove instrumental to our upper bounds. In order to do so, we first must introduce some of the notions they require, starting with that of a densifier:

Definition 8**.**

Fix a function $\gamma(n,1/\varepsilon,1/\delta)$ taking values in $(0,1]$ and two classes $\mathcal{C},\mathcal{C}^{\prime}$ of $n$-variate Boolean functions from $\mathbb{R}^{n}$ to $\{-1,1\}$. An algorithm $\mathcal{A}_{\rm den}^{(\mathcal{C},\mathcal{C}^{\prime})}$ is said to be a $\gamma$-densifier for $\mathcal{C}$ using $\mathcal{C}^{\prime}$ under $\mathcal{D}$ if it has the following behavior: For every $\varepsilon,\delta\in(0,1]$, every $1/2^{n}\leq\widehat{p}\leq 1$, and every $f\in\mathcal{C}$, given as input $\varepsilon,\delta,\widehat{p}$ and a set of independent samples from $\mathcal{D}_{f}$, the following holds. Let $p:=\operatorname{{\bf Pr}}_{\bm{x}\leftarrow\mathcal{D}}\left[\,f(\bm{x})=1\,\right]$. If $p\leq\widehat{p}<(1+\varepsilon)p$, then with probability at least $1-\delta$ the algorithm outputs a function $g\in\mathcal{C}^{\prime}$ such that: (a) $\operatorname{{\bf Pr}}_{\bm{x}\leftarrow\mathcal{D}_{f}}\left[\,g(\bm{x})=1\,\right]\geq 1-\varepsilon$, and (b) $\operatorname{{\bf Pr}}_{\bm{x}\leftarrow\mathcal{D}_{g}}\left[\,f(\bm{x})=1\,\right]\geq\gamma(n,1/\varepsilon,1/\delta)$.

The intuition behind this definition is that by (a) almost all of the satisfying assignments of $f$ are also satisfying assignments of $g$, and by (b) the satisfying assignments of $f$ are “dense” in $g^{-1}(1)$ (at least a $\gamma$ fraction under ${\cal D}$). We will sometimes make explicit the parameters of the definition, and write that such an algorithm is an $(\varepsilon,\gamma,\delta)$-densifier.

The second notion we need is that of the statistical query (SQ) learning model from [Kea98], a restriction of the PAC learning model which only allows the learning algorithm to obtain, via a statistical query oracle ${\sf STAT}(f,\mathcal{D})$, estimates of the form $\mathbb{E}_{\bm{x}\leftarrow\mathcal{D}}[\chi(\bm{x},f(\bm{x}))]$ to within an additive $\tau$, where $\chi\colon\mathbb{R}^{n}\times\{-1,1\}\to\{-1,1\}$ and $\tau\in(0,1]$ constitute the query $(\chi,\tau)$, and $f\in\mathcal{C}$ is the unknown concept. As in the PAC model, the algorithm must output a hypothesis $g$ such that $\operatorname{{\bf Pr}}_{\bm{x}\leftarrow\mathcal{D}}\left[\,f(\bm{x})\neq g(\bm{x})\,\right]\leq\varepsilon$ with probability at least $1-\delta$; such an algorithm, referred to as an $(\varepsilon,\delta)$-SQ learning algorithm for $\mathcal{C}$, is said to be efficient if it makes a number of queries polynomial in $n$, $1/\varepsilon$, and $\log(1/\delta)$, all with accuracy $\tau=\mathrm{poly}(1/n,\varepsilon)$.

We will also need the standard definitions of approximate counting and approximate generation, slightly adapted to our setting in which the relevant background distribution is some other distribution than the uniform distribution over $\{0,1\}^{n}$. Since it is more straightforward, we begin with the notion of approximate counting that we will require:

Definition 9**.**

Let $\mathcal{C}$ be a class of $n$-variate Boolean functions mapping $\mathbb{R}^{n}$ to $\{-1,1\}$. A randomized algorithm $\mathcal{A}_{\rm count}^{(\mathcal{C})}$ is said to be an efficient approximate counting algorithm for $\mathcal{C}$ under $\mathcal{D}$ if, for any $\varepsilon,\delta\in(0,1]$ and any $f\in\mathcal{C}$, on input $\varepsilon,\delta$ and $f\in\mathcal{C}$ and given oracle access to $\mathcal{D}$, it runs in time $\mathrm{poly}(n,1/\varepsilon,\log(1/\delta))$ and with probability at least $1-\delta$ outputs a value $\widehat{p}$ such that

[TABLE]

Taking ${\mathcal{D}}$ to be the uniform distribution over $\{0,1\}^{n}$, this corresponds precisely to the usual notion of approximate counting which was used in [DDS15a].

There is an issue which arises in defining approximate generation algorithms in our setting which we now discuss (and explain how to handle). We begin with the following definition:

Definition 10**.**

Let $\mathcal{C}$ be a class of $n$-variate Boolean functions mapping $\mathbb{R}^{n}$ to $\{-1,1\}$. A randomized algorithm $\mathcal{A}_{\rm gen}^{(\mathcal{C})}$ is said to be an efficient strong approximate generation algorithm for $\mathcal{C}$ under $\mathcal{D}$ if, for any $\varepsilon\in(0,1]$ and $f\in\mathcal{C}$, there is a distribution $\widehat{\mathcal{D}}_{f,\varepsilon}$ supported on $f^{-1}(1)$ with

[TABLE]

for every $x\in f^{-1}(1)$, such that for any $\delta\in(0,1]$, on input $\varepsilon,\delta$, and $f\in\mathcal{C}$, and given oracle access to $\mathcal{D}$, the algorithm runs in time $\mathrm{poly}(n,1/\varepsilon,\log(1/\delta))$ and either outputs a point $x\in f^{-1}(1)$ exactly distributed according to $\widehat{\mathcal{D}}_{f,\varepsilon}$, or outputs $\bot$. Moreover, the probability that it outputs $\bot$ is at most $\delta$.

Taking the distribution ${\cal D}$ in Definition 10 to be the uniform distribution over $\{-1,1\}^{n}$, we recover precisely the usual notion of approximate generation which was used in [DDS15a]. While this compatibility with [DDS15a] is an attractive feature, it is not difficult to see that the requirement that $\widehat{\mathcal{D}}_{f,\varepsilon}$ put approximately the right amount of weight (in a multiplicative sense) on every $x\in f^{-1}(1)$ is overly demanding. If ${\cal D}_{f}$ puts only an extremely small amount of weight on a point $x$, it may be impossible for an efficient algorithm to very precisely match this extremely small amount of weight; moreover, since the weight on $x$ is so tiny, as long as a purported approximate generation algorithm puts only a tiny amount of weight on $x$, it should not matter whether or not the tiny amount it puts is multiplicatively accurate. We thus introduce the following relaxed notion of approximate generation, which we call “weak approximate generation”:

Definition 11**.**

Let $\mathcal{C}$ be a class of $n$-variate Boolean functions mapping $\mathbb{R}^{n}$ to $\{-1,1\}$. A randomized algorithm $\mathcal{A}_{\rm gen}^{(\mathcal{C})}$ is said to be an efficient weak approximate generation algorithm for $\mathcal{C}$ under $\mathcal{D}$ if, for any $\tau\in(0,1]$ and any $f\in\mathcal{C}$, there is a distribution ${\mathcal{D}}^{\prime}_{f,\tau}$ with

[TABLE]

such that on input $\tau$, and $f\in\mathcal{C}$, and given oracle access to $\mathcal{D}$, the algorithm runs in time $\mathrm{poly}(n,\log(1/\tau))$ and outputs a point $x\in f^{-1}(1)$ distributed according to ${\mathcal{D}}^{\prime}_{f,\tau}$.

2.2 The algorithmic result we use

With the above notions in hand, we are ready to state the algorithmic result that we will rely on:

Theorem 12** (based on [DDS15a, Theorem 3.1]).**

Let $\mathcal{C},\mathcal{C}^{\prime}$ be classes of $n$-variate Boolean functions mapping $\mathbb{R}^{n}$ to $\{-1,1\}$. Suppose that

• •

$\mathcal{A}_{\rm den}^{(\mathcal{C},\mathcal{C}^{\prime})}$* is an $(\varepsilon,\gamma,\delta)$-densifier for $\mathcal{C}$ using $\mathcal{C}^{\prime}$ under $\mathcal{D}$ running in time $T_{\rm den}(n,1/\varepsilon,1/\delta)$.*

• •

$\mathcal{A}_{\rm count}^{(\mathcal{C}^{\prime})}$* is an $(\varepsilon,\delta)$-approximate counting algorithm for $\mathcal{C}^{\prime}$ under $\mathcal{D}$ running in time $T_{\rm count}(n,1/\varepsilon,1/\delta)$.*

• •

$\mathcal{A}_{\rm gen}^{(\mathcal{C}^{\prime})}$* is an $(\varepsilon,\delta)$-weak approximate generation algorithm for $\mathcal{C}^{\prime}$ under $\mathcal{D}$ running in time $T_{\rm gen}(n,1/\varepsilon,1/\delta)$.666Actually, the theorem only requires that $\mathcal{A}_{\rm count}^{(\mathcal{C}^{\prime})}$ and $\mathcal{A}_{\rm gen}^{(\mathcal{C}^{\prime})}$ succeed as long as $f\in\mathcal{C^{\prime}}$ satisfies $\operatorname{{\bf Pr}}_{x\leftarrow\mathcal{D}}[f(x)=1]\geq 2^{-\mathrm{poly}(n)}$.*

• •

$\mathcal{A}_{\rm SQ}^{(\mathcal{C})}$* is an $(\varepsilon,\delta)$-SQ learning algorithm for $\mathcal{C}$ such that: $\mathcal{A}_{\rm SQ}^{(\mathcal{C})}$ runs in time $t_{1}(n,1/\varepsilon,1/\delta)$, $t_{2}(n)$ is the maximum time needed to evaluate any query provided to ${\sf STAT}$, and $\tau(n,1/\varepsilon)$ is the minimum value of the tolerance parameter ever provided to ${\sf STAT}$ in the course of $\mathcal{A}_{\rm SQ}^{(\mathcal{C})}$’s execution.*

Then there exists a distribution learning algorithm $\mathcal{A}^{\mathcal{C}}$ for $\mathcal{C}$ under $\mathcal{D}$. The running time of $\mathcal{A}^{\mathcal{C}}$ is polynomial in $T_{\rm den}(n,1/\varepsilon,1/\delta)$, $T_{\rm count}(n,1/\varepsilon,1/\delta)$, $T_{\rm gen}(n,1/\varepsilon,1/\delta)$, $t_{1}(n,1/\varepsilon,1/\delta)$, $t_{2}(n)$, and $1/\tau(n,1/\varepsilon)$, provided that $T_{\rm den}(\cdot),T_{\rm count}(\cdot),T_{\rm gen}(\cdot),t_{1}(\cdot),t_{2}(\cdot),1/\tau(\cdot)$ are polynomial in their parameters.

The above statement is almost verbatim from [DDS15a], with the difference that (i) we apply it to our setting of continuous distributions and Boolean functions over $\mathbb{R}^{n}$, and (ii) the third bullet uses the notion of a weak approximate generation algorithm rather than a strong one. Theorem 12 can be established in two stages. First, it is straightforward to check by inspection of the proof of [DDS15a] that the proof of their Theorem 3.1 goes through essentially unchanged to establish the variant of the above theorem in which we substitute “strong” for “weak” in the third bullet. (We note that the proof of Theorem 3.1 of [DDS15a] requires a multiplicatively accurate evaluation oracle for each hypothesis distribution constructed in the course of the algorithm’s execution, and that such oracles can be obtained in our setting since by our assumptions we have an approximate counting algorithm for the class $\mathcal{C}^{\prime}$.) For the second stage, we claim that the variant of the above theorem with “strong” in place of “weak” easily implies Theorem 12 as stated above (with “weak” in the third bullet). To see this, observe that if $s$ draws are made from a distribution ${\cal D}^{\prime}_{f,\tau}$ which has variation distance at most $\tau$ from ${\cal D}_{f}$, then the variation distance between the distribution of that sample of $s$ draws and the corresponding distribution (over samples of $s$ draws) from ${\mathcal{D}}_{f}$ is at most $s\tau.$ Hence any procedure $A$ (such as the distribution learning algorithm $\mathcal{A}^{\mathcal{C}}$) which makes $s$ calls to a strong approximate generation algorithm can instead be run using a weak approximate generation algorithm, and if the $\tau$-parameter of the weak approximate generation algorithm is set to be $\delta/s$, this will decrease the success probability of the procedure $A$ by at most an additive $\delta.$

Miscellaneous notation and terminology. We write $B^{1}(z,r)$ to denote the $\ell_{1}$-ball of radius $r$ in $\mathbb{R}^{n}$ around a point $z$, and write $B^{2}(z,r)$ to denote the $\ell_{2}$-ball.

3 A distribution learning algorithm for LTFs with respect to log-concave background distributions

3.1 Sampling and integration over log-concave distributions

The ability to efficiently sample from log-concave distributions, and to approximate various quantities such as the normalizing factor of a log-concave function, is crucial for our upper bound result Theorem 5. As mentioned in the introduction, there is an extensive literature on these problems. In this subsection we state and explain the specific results we shall rely on, and prove that our assumptions imply that we can indeed use them in our setting. In particular, the key take-away result from this subsection is the following crucial lemma:

Lemma 3.1**.**

Let $\tilde{\mu}\colon\mathbb{R}^{n}\rightarrow[0,\infty)$ be a log-concave measure which is assumed to satisfy conditions (C1), (C2) and (C3) from Section 2, and let $\mu:=\tilde{\mu}/{\lVert\tilde{\mu}{\rVert}}_{1}$ be the corresponding probability distribution. There is an algorithm with the following performance guarantee: Given access to an evaluation oracle for $\tilde{\mu}$, a halfspace $g\colon\mathbb{R}^{n}\rightarrow\{-1,1\}$, a confidence parameter $\delta$ and an error parameter $\varepsilon>1/2^{o(n)}$ such that $\operatorname{{\bf Pr}}_{\bm{x}\leftarrow\mu}[g(\bm{x})=1]\geq 2^{-n}$, it runs in time $\mathrm{poly}(n,1/\varepsilon,\log(1/\delta))$ and with probability $1-\delta$ outputs a number $\widehat{p}$ such that

[TABLE]

Let $\mu_{g}$ denote the distribution $\mu$ conditioned on $g(x)=1$. Under the above assumptions, there is also a randomized algorithm which runs in time $\mathsf{poly}(n,\log(1/\varepsilon))$ and samples from a distribution $\nu$ such that $\|\nu-\mu_{g}\|_{1}\leq\varepsilon$.

3.1.1 Setup for the proof of Lemma 3.1

We first observe that we can apply Theorem 38 (see Appendix B) to the log-concave measure $\tilde{\mu}$ to obtain a new measure $\tilde{\mu}_{1}$ which is a “smoothed” version of $\tilde{\mu}$. As detailed in Theorem 38, the distribution $\mu_{1}$ corresponding to $\tilde{\mu}_{1}$ satisfies $\|\mu-\mu_{1}\|_{1}\leq 2^{-3n/2}$, so $\mu$ and $\mu_{1}$ are statistically extremely close. Let $\mu_{g}$ (resp. $\mu_{1g}$) denote the distribution $\mu$ (resp. $\mu_{1}$) conditioned on $g=1$. Since $\operatorname{{\bf Pr}}_{\bm{x}\leftarrow\mu}[g(\bm{x})=1]\geq 2^{-n}$, it follows that $\|\mu_{g}-\mu_{1g}\|_{1}\leq 2^{-n/2}$. Since we only work with error parameters $\varepsilon\geq 1/2^{o(n)}$, the algorithms we consider only draw at most $2^{o(n)}$ many samples from $\mu_{g}$, and hence with very high probability sampling from $\mu_{1g}$ is identical to sampling from $\mu_{g}$. Thus, from now on, we will assume the following (see Theorem 38):

Claim \theclaim.

The log-concave measure $\tilde{\mu}$ satisfies the following conditions.

For $x\not\in B^{2}(0,n^{C})$, $\tilde{\mu}(x)=0$. 2. 2.

The resulting density $\mu$ is such that for all $x\in B^{2}(0,n^{C})$, $\mu(x)\geq 2^{-2n^{C}-5n-3}\cdot n^{-2Cn}$.

Lemma 3.1 is proved by leveraging results of [LV06]. We first recall two assumptions from [LV06] which suffice for their results to go through. Their algorithms deal with an unknown log-concave function $\tilde{\mu}\colon\mathbb{R}^{n}\to[0,\infty)$ (and the associated probability density function $\mu:=\tilde{\mu}/{\lVert\tilde{\mu}{\rVert}}_{1}$), and are given

1. (LV1)

an evaluation oracle that returns the value of $\tilde{\mu}$ at any given $x\in\mathbb{R}^{n}$, as well as two values $R,r>0$ such that (i) the variance of $\mu$ in every direction is at most $R^{2}$; and (ii) if the level set $L(c):=\left\{\;x\in\mathbb{R}^{n}\;\colon\;\tilde{\mu}(x)\geq c\;\right\}$ satisfies $\mu(L(c))\geq 1/8$, then $L(c)$ contains a ball of radius $r$; and 2. (LV2)

a random “starting point” $\bm{x}_{0}\in\mathbb{R}^{n}$ drawn from a probability distribution $\sigma$, along with a bound $M\geq 1$ such that $\frac{d\sigma}{d\mu}\leq M$.

Remark 13**.**

If condition (LV1) holds for $R$ and $r$ such that $R/r=O(\sqrt{n})$, then the log-concave measure $\tilde{\mu}$ (and the resulting distribution $\mu$) is said to be well-rounded.

Theorem 1.1 in [LV06] states that under the conditions (LV1) and (LV2), there is an efficient algorithm to approximately sample from the density $\mu$:

Theorem 14** (Theorem 1.1, [LV06]).**

There is an algorithm $A_{\text{LV-sampler}}$ which, under conditions (LV1) and (LV2), given an error parameter $\varepsilon>0$, has the following properties:

The running time of $A_{\text{LV-sampler}}$ is $\mathrm{poly}(n,R/r,\log(1/\varepsilon),\log M)$. 2. 2.

The output distribution of the algorithm (denoted by $\nu$) satisfies $\|\mu-\nu\|_{1}\leq\varepsilon$.

Note that the algorithm in Theorem 14 runs in polynomial time provided that $\varepsilon\geq 2^{-n^{O(1)}}$, $M\leq 2^{n^{O(1)}}$, and $R/r\leq\mathrm{poly}(n).$ Moreover, when the distribution $\mu$ is well-rounded, then the next theorem (Theorem 1.3 in [LV06]) also gives an efficient algorithm to compute $\int\tilde{\mu}(x)$.

Theorem 15**.**

There is an algorithm $A_{\text{LV-integral}}$ which, for a well-rounded log-concave function $\tilde{\mu}$, under conditions (LV1) and (LV2), given access to an error parameter $\varepsilon>0$, outputs a number $Q$ such that

[TABLE]

The running time of the algorithm is $\mathrm{poly}(n/\varepsilon)$.

We stress that both Theorem 15 and Theorem 14 rely on $R/r$ being not too large. While this may not always be the case for the distribution $\mu$ to begin with, it is possible to achieve this via a linear transformation, as we discuss next.

Definition 16**.**

Let $T\colon\mathbb{R}^{n}\to\mathbb{R}^{n}$ be an invertible linear transformation, let $z\in\mathbb{R}^{n}$, and let $\mu$ be any measure supported on $\mathbb{R}^{n}$. We define the measure $\mu_{T,z}$ which is given by

[TABLE]

We observe that if $\mu$ is a probability density, then $\mu_{T,z}$ is the density obtained by sampling $\bm{x}\leftarrow\mu$ and applying the linear transformation $x\mapsto Tx+z$ to $\bm{x}$. Since linear transformations preserve log-concavity, if $\mu$ is log-concave then $\mu_{T,z}$ is also log-concave.

We further observe that for any log-concave measure $\tilde{\mu}$, any $z\in\mathbb{R}^{n}$, and any invertible matrix $T$, we have

[TABLE]

Moreover, given an efficient algorithm that samples from a distribution that is $\varepsilon$-close to the distribution $\mu_{T,z}$, there is also an efficient algorithm to sample from a distribution that is $\varepsilon$-close to distribution $\mu$.

With the above observations in hand, a strategy to prove Lemma 3.1 naturally suggests itself. Given oracle access to a log-concave measure $\tilde{\mu}$ and a halfspace $g$ satisfying the conditions of Lemma 3.1, we will compute a linear transformation $T\colon\mathbb{R}^{n}\rightarrow\mathbb{R}^{n}$ and a random starting point $\bm{z}\in\mathbb{R}^{n}$ such that the density $\tilde{\mu}_{T,z}$ satisfies LV1 with $R/r=O(\sqrt{n})$. In particular, we will prove the following lemma:

Lemma 3.2**.**

Let $\tilde{\mu}\colon\mathbb{R}^{n}\to[0,\infty)$ be a log-concave measure and $g$ be a halfspace satisfying the hypothesis of Lemma 3.1. There is a polynomial time algorithm which, given evaluation oracle access to $\tilde{\mu}$ and a halfspace $g$, computes an invertible linear transformation $T\colon\mathbb{R}^{n}\to\mathbb{R}^{n}$ and a $z\in\mathbb{R}^{n}$ such that the density $({\mu}_{T,z})_{g}$ satisfies LV1 with $R/r=O(\sqrt{n})$.

3.1.2 Proof of Lemma 3.1 assuming Lemma 3.2

We now explain the details of how Lemma 3.2 implies Lemma 3.1. To do this, we need to recall the following result from [KLS97]:

Theorem 17** ([KLS97]).**

Given a separation oracle for a convex body $K\subseteq\mathbb{R}^{n}$, there is an algorithm $A_{\text{KLS-sampler}}$ which makes $\tilde{O}(n^{5})$ oracle calls and produces a uniformly random sample from $K$.

Proof of Lemma 3.1 assuming Lemma 3.2.

Let us define the set $\mathcal{A}_{g}\subset\mathbb{R}^{n}$ to be ${\cal A}_{g}=\{x\in B^{2}(0,n^{d}):g(x)=1\}$, and let us define $\sigma$ to be the uniform distribution over $\mathcal{A}_{g}$. We observe that we can efficiently sample from the distribution $\sigma$ because the set $\mathcal{A}_{g}$ has an separation oracle (recall that in the setting of Lemma 3.1 we are given $g$) and thus we can sample from $\mathcal{A}_{g}$ using the algorithm $A_{\text{KLS-sampler}}$ from Theorem 17. Let $V$ denote the volume of the set $\mathcal{A}_{g}$, so $\sigma(x)=1/V$ for all $x\in\mathcal{A}_{g}$. Since $\operatorname{{\bf Pr}}_{\bm{x}\leftarrow\mu}[g(\bm{x})=1]\geq 2^{-n}$ and $\mu(x)\leq 2^{n}$ for all $x$ (by condition C2), it follows that $V\geq 2^{-2n}$, and consequently for all $x\in\mathcal{A}_{g}$, we have $\sigma(x)\leq 2^{2n}$.

By Section 3.1.1, $\mu(x)\geq 2^{-2n^{C}-5n-3}\cdot n^{-2Cn}$ for every $x\in B^{2}(0,n^{C})$. Since $\operatorname{{\bf Pr}}_{\bm{x}\leftarrow\mu}[g(\bm{x})=1]\geq 2^{-n}$, it follows that $\mu_{g}(x)\geq 2^{-2n^{C}-7n-3}\cdot n^{-2Cn}$. Consequently, we have that for every $x\in\mathcal{A}_{g}$,

[TABLE]

Hence for any invertible linear transformation $T\colon\mathbb{R}^{n}\rightarrow\mathbb{R}^{n}$ and any $z\in\mathbb{R}^{n}$, we also get

[TABLE]

In particular, if we apply the linear transformation $T$ and $z$ by applying Lemma 3.2 to the log-concave measure $\tilde{\mu}$ and the halfspace $g$, then we satisfy LV1 and LV2 with $R/r=O(\sqrt{n})$ and $M=2^{2n^{C}+9n+3}\cdot n^{2Cn}$, taking the point $\bm{x}_{0}$ specified in LV2 to be a random point sampled from $\sigma$. We can now apply Theorem 15 and Theorem 14 to obtain the desired conclusion of Lemma 3.1. ∎

3.1.3 Proof of Lemma 3.2

In the rest of this section we prove Lemma 3.2. Towards this end, we first recall a useful definition:

Definition 18**.**

Let $\mu$ be a log-concave distribution supported on $\mathbb{R}^{n}$. For $\beta>0$, $\mu$ is said to be $\beta$-isotropic if for every unit vector $u\in\mathbb{R}^{n}$, we have

[TABLE]

In other words, the variance of $\mu$ in every direction is between $1/\beta$ and $\beta$. A log-concave measure $\tilde{\mu}$ is said to be $\beta$-isotropic if the corresponding distribution $\mu$ is $\beta$-isotropic.

Lovász and Vempala showed that if a log-concave distribution is $1$-isotropic, then it is well-rounded (Lemma 5.13, [LV07]). However, it is easy to see by inspection of their proof that if a log-concave distribution is $\beta$-isotropic for any absolute constant $\beta>1$, then it is also well-rounded. We record this here:

Lemma 3.3**.**

If a log-concave distribution $\mu$ is $\beta$-isotropic for an absolute constant $\beta>1$, then it is well-rounded.

We need two more ingredients to prove Lemma 3.2. The first is a result of Lovász and Vempala which shows that given a maximizer of a log-concave measure, the measure can efficiently be brought to an $O(1)$-isotropic position.

Theorem 19** ([LV07]).**

Given evaluation oracle access to a full-dimensional log-concave measure $\tilde{\nu}$ and a point $y$ such that $\tilde{\nu}(y)\geq(1-2^{-n})\cdot\max_{x}\tilde{\nu}(x)$, there is an algorithm which makes $\tilde{O}(n^{4})$ oracle calls to $\tilde{\nu}$ and computes an invertible linear transformation $T$ and a point $z\in\mathbb{R}^{n}$ such that the transformed measure $\tilde{\nu}_{T,z}$ (defined as in Definition 16) is $2$-isotropic.

We remark that the theorem statement of Theorem 19 in [LV07] stipulates that the point $y$ be an exact maximizer, as opposed to a near-maximizer; however, by inspecting the proof one can verify that having a near-maximizer (in the sense provided by our theorem statement) also suffices to find the linear transformation $T$ and point $z.$777This was confirmed in personal communication with Vempala [Vem18].

The second ingredient we will use is the following theorem from [LSV18] (though weaker results might also suffice):

Theorem 20** ([LSV18, Theorem 1]).**

Let $K$ be a convex set specified by a membership oracle, let $w\in\mathbb{R}^{n}$, and let $0<r^{\prime}<R^{\prime}$ be numbers such that $B^{2}(w,r^{\prime})\subseteq K\subseteq B^{2}(w,R^{\prime})$. For any convex function $f\colon\mathbb{R}^{n}\to\mathbb{R}$ given by an evaluation oracle and any $\varepsilon>0$, there is a randomized algorithm that with constant probability computes a point $z\in B^{2}(K,\varepsilon)$ such that

[TABLE]

using n^{2}\log^{O(1)}\big{(}\frac{n\cdot R^{\prime}}{\varepsilon\cdot r^{\prime}}\big{)} oracle calls and n^{3}\log^{O(1)}\big{(}\frac{n\cdot R^{\prime}}{\varepsilon\cdot r^{\prime}}\big{)} arithmetic operations.

Now we are ready to prove Lemma 3.2:

Proof of Lemma 3.2.

By applying Lemma 3.3, it suffices to compute $T,z$ such that the distribution $({\mu}_{T,z})_{g}$ is $2$-isotropic. We start by observing that $\tilde{\mu}_{g}(x)$ is proportional to $\tilde{\mu}(x)\cdot\mathbf{1}[g(x)=1]$, so in the setting of Lemma 3.2 we have evaluation oracle access to a measure proportional to $\tilde{\mu}_{g}$. By Theorem 19, it suffices to find a point $y\in\mathbb{R}^{n}$ such that $\tilde{\mu}_{g}(y)\geq(1-2^{-n})\cdot\max_{x\in\mathcal{A}_{g}}\tilde{\mu}_{g}(x)$.

To find the required point $y$, we will use Theorem 20. Note that since $\tilde{\mu}_{g}$ is log-concave, the function $-\log\tilde{\mu}_{g}$ is convex, so minimizing $h(x)=-\log\tilde{\mu}_{g}(x)$ is equivalent to maximizing $\tilde{\mu}_{g}(x)$. We now provide the details. Recalling that $\|\mu\|_{\infty}\leq 2^{n}$ and $\operatorname{{\bf Pr}}_{\bm{x}\leftarrow\mu}[g(\bm{x})=1]\geq 2^{-n}$, we get that $\|\mu_{g}\|_{\infty}\leq 2^{2n}$; combining this with the lower bound on $\mu_{g}(x)$ established in Section 3.1.2, we get that

[TABLE]

for all $x\in B^{2}(0,n^{C})$. We take the set $K$ of Theorem 20 to be $K=\mathcal{A}_{g}$, and we observe that by the above bounds on $\mu_{g}$, we have

[TABLE]

We further observe that we clearly have both a separation oracle and a membership oracle for the set $K$. We use this separation oracle with Theorem 17 to generate a point $\bm{w}$ which is uniformly sampled from $K$. We will now show that setting

[TABLE]

we have $B^{2}(\bm{w},r^{\prime})\subseteq K\subseteq B^{2}(\bm{w},R^{\prime})$. The second inclusion is clear (that for the above $R^{\prime}$, we have $K\subseteq B^{2}(w,R^{\prime})$). For the first inclusion we recall the following fact from high-dimensional convex geometry (which follows from the existence of higher-dimensional Crofton formulas, see, e.g., [sur16]):

Fact \thefact.

Let $K$ be any convex body contained in a ball of radius $\rho=n^{C}$. Then the surface area of $K$ is at most the surface area of the sphere of radius $\rho$ (which is bounded by $2\pi^{n-1}\rho^{n-1}$).

We will now use this to bound the volume of the set of the points at a distance $r^{\prime}$ from the boundary of $\mathcal{A}_{g}$. To do this, for any convex body $K$, define $K_{t}=\{x:\mathsf{dist}(x,K)\leq t\}$. Observe that $K_{t}$ is a convex body for any $K$ and $t$. The following relation is well-known:

[TABLE]

Here $\mathsf{surf}(K)$ is the surface area of $K$. Thus, it follows that for any $t\geq 0$,

[TABLE]

Let $A_{g,t}=\{x:\mathsf{dist}(x,A_{g})\leq t\}$. Applying Fact 3.1.3 to $A_{g,t}$, we have $\mathsf{surf}(A_{g,t})\leq 2\pi^{n-1}\cdot(n^{C}+t)^{n-1}$. This is because if $A_{g}$ is contained in a ball of radius $n^{C}$, then $A_{g,t}$ is contained in a ball of radius $n^{C}+t$. Combining this with (2) applied to the body $A_{g}$,

[TABLE]

Using the fact that $r^{\prime}\leq n^{C}$, we have that $\mathsf{vol}(A_{g,r^{\prime}})-\mathsf{vol}(A_{g})\leq 2\pi^{n-1}(2n)^{(n-1)C}\cdot r^{\prime}$. Thus, the total volume of the set of points at a distance $r^{\prime}$ from the boundary of $\mathcal{A}_{g}$ is at most $2\pi^{n-1}(2n)^{(n-1)C}\cdot r^{\prime}$. By plugging in the value of $r^{\prime}$ from above, this volume is at most $2^{-3n}$. Since $\mathsf{Vol}(\mathcal{A}_{g})\geq 2^{-2n}$ (see  Section 3.1.2), this means that with probability $1-2^{-n}$, the point $\bm{w}$ satisfies $B^{2}(w,r^{\prime})\subseteq K\subseteq B^{2}(w,R^{\prime})$.

We now set $\varepsilon$ to be

[TABLE]

We apply Theorem 20 with $w$, $R^{\prime}$, $r^{\prime}$ and $\varepsilon$ as above to get a point $z$ such that

[TABLE]

Thus, $\tilde{\mu}_{g}(z)\geq e^{-2^{-2n-1}}\cdot\max_{x\in K}\tilde{\mu}_{g}(x)$, which implies that $\tilde{\mu}_{g}(z)\geq(1-2^{-n})\max_{x\in K}\tilde{\mu}_{g}(x)$. With this point $z$ in hand, we can apply Theorem 19 to bring $\tilde{\mu}_{g}$ to $2$-isotropic position. This finishes the proof. ∎

3.2 An algorithm for LTFs under well-behaved log-concave distributions

In this subsection we prove Theorem 5. We do this by applying Theorem 12, where both the classes $\mathcal{C}$ and $\mathcal{C}^{\prime}$ are taken to be the class $\mathsf{PTF}^{n}_{1}$ of linear threshold functions over $\mathbb{R}^{n}$. Thus what is needed is to establish the existence of each of the four algorithmic ingredients — a densifier, an approximate counting algorithm, a weak approximate generation algorithm, and an SQ-learning algorithm — required by Theorem 12. (Some of these will be reused in the proof of Theorem 6 as well.)

Several of these ingredients are provided for us in the literature. An SQ-learning algorithm for the class of linear threshold functions over $\mathbb{R}^{n}$ was given in [BFKV97]:

Theorem 21**.**

There is a distribution-independent SQ learning algorithm $\mathcal{A}_{\rm SQ}^{\mathsf{LTF}}$ for the class of linear threshold functions over $\mathbb{R}^{n}$ that has running time $\mathrm{poly}(n,1/\varepsilon,\log(1/\delta))$, uses at most $\mathrm{poly}(n)$ time to evaluate each query and requires tolerance of its queries no more than $\tau=1/\mathrm{poly}(n,1/\varepsilon)$.

An approximate counting algorithm for linear threshold functions under well-behaved log-concave distributions is given by Lemma 3.1, and this lemma also gives a weak approximate generation algorithm. Thus, it remains only to establish the existence of an efficient densifier for linear threshold functions under log-concave background distributions.

Construction of a densifier for $\mathsf{PTF}^{n}_{1}$ under well-behaved log-concave distributions. To explain the construction of the densifier, we will need to recall the notion of an online learning algorithm. An online algorithm $\mathcal{A}$ for a concept class $\mathcal{C}$ operates in a sequential model of learning: at every stage, the algorithm is presented with an unlabeled point $x$ and is asked to predict the binary value $f(x)$ where $f$ is the target function. After the prediction, the true value of $f(x)$ is revealed. $\mathcal{A}$ is said to have a mistake bound of $M$ if the maximum number of incorrect predictions made by the algorithm is $M$ on any sequence of unlabeled examples and any target function $f\in\mathcal{C}$.

We will use the following celebrated result from [MT94], which gives an online learning algorithm for $\mathsf{PTF}^{n}_{1}$ if the example points come from a discrete grid:

Theorem 22**.**

There is an online learning algorithm (which we call $\mathcal{M}\mathcal{T}_{n,L}$) for $\mathsf{PTF}^{1}_{n}$ over the domain $[L]^{n}$ with the following guarantee: $\mathcal{M}\mathcal{T}_{n,L}$ has a mistake bound of $M(L,n):=O(n^{2}(\log L+\log n))$ and runs in time $\mathrm{poly}(n,\log L)$ at each stage. Further, at every stage of its execution, $\mathcal{M}\mathcal{T}_{n,L}$ maintains a (weights based) representation of a LTF which correctly labels all the examples received so far.

We now state our main theorem establishing the existence of densifiers for $\mathsf{PTF}^{1}_{n}$ when the ambient distribution is a well-behaved log-concave distribution:

Theorem 23**.**

Let $\mathcal{D}$ be any log-concave density satisfying conditions C1, C2 and C3. There is an algorithm $\mathcal{A}^{\mathsf{LTF}}_{\mathsf{den}}$ which is an $(\varepsilon,\gamma,\delta)$-densifier for $\mathsf{PTF}^{n}_{1}$ using $\mathsf{PTF}^{n}_{1}$ under $\mathcal{D}$ where $\gamma=1/\mathsf{poly}(n).$ Algorithm $\mathcal{A}^{\mathsf{LTF}}_{\mathsf{den}}$ has running time $\mathsf{poly}(n,1/\varepsilon,\log(1/\delta))$.

The proof of Theorem 23 is a modification of the analogous proof in [DDS15a] (Theorem 11 in the full version) where the background distribution $\mathcal{D}$ is uniform over $\{-1,1\}^{n}$, so, instead of giving a full proof, we limit ourselves to a proof sketch. (The key new issue that needs to be dealt with is the fact that mistake-bound guarantees do not hold in a completely unrestricted setting of learning over continuous domains. At a high level we surmount this issue by exploiting specific properties of our well-behaved log-concave distributions, such as the fact that such a density cannot be too concentrated in a very small region.)

3.2.1 Proof sketch of Theorem 23

We begin by recalling that to show a function $g$ is a densifier for $f$, it suffices to establish two properties:

(a)

Agreement: $\operatorname{{\bf Pr}}_{\bm{x}\leftarrow\mathcal{D}_{f}}\left[\,g(\bm{x})=1\,\right]\geq 1-\varepsilon$, and

(b)

Density: $\operatorname{{\bf Pr}}_{\bm{x}\leftarrow\mathcal{D}}\left[\,g(\bm{x})=1\,\right]\leq(1/\gamma)\operatorname{{\bf Pr}}_{\bm{x}\leftarrow\mathcal{D}}\left[\,f(\bm{x})=1\,\right]$.

A densifier for LTFs in the uniform-distribution [DDS15a] setting. We first recall the algorithm to construct a densifier from [DDS15a] and give a high level overview of its proof of correctness (Theorem 11 in [DDS15a]). Note that [DDS15a] constructs a densifier for $\mathsf{PTF}^{n}_{1}$ where the domain is $\{-1,1\}^{n}$ (or equivalently, $[2]^{n}$) and the background distribution $\mathcal{D}$ is uniform on $\{-1,1\}^{n}$. In the following, the value of $\gamma$ is chosen to be a suitably small $1/\mathrm{poly}(n)$ quantity (small compared to $1/M$ as defined below).

Sample $N_{+}=\Theta((n^{2}+\log(1/\delta))/\varepsilon^{2})$ points $\bm{x}_{1},\ldots,\bm{x}_{N_{+}}$ where each $\bm{x}_{i}$ is drawn from ${\cal D}_{f}$ (i.e., ${\cal D}$ conditioned on $f(\bm{x}_{i})=1$). 2. 2.

Define the set $\mathcal{S}^{+}:=\{\bm{x}_{1},\ldots,\bm{x}_{N_{+}}\}$. Let $M=O(n^{2}\log n)$ and $i=0$. 3. 3.

Initialize the online algorithm $\mathcal{M}\mathcal{T}_{n,L}$ with $L=2$ and let $h_{0}$ be the initial hypothesis. 4. 4.

If $i=M$, output $h_{i}$ and terminate. 5. 5.

If there exists $x\in S^{+}$ such that $h_{i}(x)=-1$, then feed such an $x$ to $\mathcal{M}\mathcal{T}_{n,L}$ along with the label $\ell_{x}=1$ for $x$. Go to Step 4. 6. 6.

Otherwise, use the approximate counting algorithm to get $\widehat{p}_{i}$, a high-accuracy approximation of $\operatorname{{\bf Pr}}_{\bm{x}\leftarrow{\cal D}}[h_{i}(\bm{x})=1]$. 7. 7.

If $\widehat{p}\geq\frac{\gamma}{2}\cdot\widehat{p}_{i}$, then terminate the procedure and output $h_{i}$. 8. 8.

Otherwise, generate $\bm{x}\leftarrow\mathcal{D}_{h_{i}}$ and feed $\bm{x}$ to $\mathcal{M}\mathcal{T}_{n,L}$ along with the label $\ell_{\bm{x}}=-1$ for $\bm{x}$. Update hypothesis $h_{i}$ using $\mathcal{M}\mathcal{T}_{n,L}$ and set $i\leftarrow i+1$. Go to Step 4.

The proof of correctness of the algorithm in [DDS15a] relies on two claims.

Claim \theclaim.

With very high probability over the draw of ${\cal S}^{+}$, (a) is satisfied by any halfspace $g\colon\{-1,1\}^{n}\to\{-1,1\}$ such that $g(x)=1$ for every $x\in S^{+}$.

Claim \theclaim.

With very high probability over the execution of the densifier, every sample $x$ and label $\ell_{x}$ fed to $\mathcal{M}\mathcal{T}_{n,L}$ is correct, i.e., $f(x)=\ell_{x}$.

(For intuition, Section 3.2.1 is argued using a simple Chernoff bound and union bound. Section 3.2.1 exploits the fact that since $\gamma$ is very small compared to $1/M$ where $M$ is the number of stages, with high probability each random point $\bm{x}\leftarrow{\cal D}_{h_{i}}$ obtained in Step 8 is indeed a negative example for $f$.) Given Section 3.2.1 and Section 3.2.1, it is not hard to show that with high probability, the output of the above algorithm terminates with a hypothesis $h_{i}$ which satisfies both (a) and (b) and thus we have a densifier.

Adapting the [DDS15a] densifier to our setting. We now return to our setting in which the background distribution $\mathcal{D}$ is log-concave and satisfies C1, C2 and C3. We change the densifier algorithm as described below.

Let $\kappa=2^{-n^{10C}}$ and let $L=1/\kappa$. For any point $x\in\mathbb{R}^{n}$, let $[x]_{\kappa}$ denote the point obtained by rounding each coordinate of $x$ to the nearest integral multiple of $\kappa$, and let $M=O(n^{2}(\log L+\log n))$ (note that this new value of $M$ is still $\mathrm{poly}(n)$). We adapt the algorithm of [DDS15a] to our setting in the following way:

Set $L$ and $M$ in the algorithm with the values specified above. 2. 2.

The background distribution is the log-concave distribution $\mathcal{D}$ satisfying C1, C2 and C3. 3. 3.

For each point $x$ fed to the learner in Steps 5 and 8, we instead feed $[x]_{\kappa}$ to the learner (so each point fed to the learner is rounded to the nearest integral multiple of $\kappa$).

These changes induce analogous changes to Sections 3.2.1 and 3.2.1, where now the points $[x]_{\kappa}$ play the role that $x$ played throughout the statements; more precisely, we state

New Version of Section 3.2.1. *With very high probability over the draw of ${\cal S}^{+}$, the bound $\operatorname{{\bf Pr}}_{\bm{x}\leftarrow{\cal D}_{f}}[g([\bm{x}]_{\kappa})=1]\geq 1-2\varepsilon$ is satisfied by any halfspace $g\colon[L]^{n}\rightarrow\{-1,1\}$ such that $g(x)=1$ for every $x\in S^{+}$. *

New Version of Section 3.2.1. *With very high probability over the execution of the densifier, every sample $[x]_{\kappa}$ and label $\ell_{x}$ fed to $\mathcal{M}\mathcal{T}_{n,L}$ is correct, i.e., $f([x]_{\kappa})=\ell_{x}$. *

We now give proof sketches of these new versions of Sections 3.2.1 and 3.2.1 in our context. (The high-level idea throughout the arguments given below is that since the rounded version $[x]_{\kappa}$ is always extremely close to $x$ and the relevant distributions are not too concentrated, they can only put a tiny amount of mass on points close to the boundary, and thus it is very unlikely that the rounding causes the label to change for any point arising in the algorithm.)

Proof Sketch of New Version of Section 3.2.1.

Let $g$ be any halfspace and define the set $K_{g}:=g^{-1}(-1)\cap f^{-1}(1)$. Observe that $K_{g}$ is an intersection of two LTFs and further that if $g$ does not satisfy (a), then

[TABLE]

We upper bound the probability $\operatorname{{\bf Pr}}_{\bm{x}\leftarrow\mathcal{D}_{f}}[\bm{x}\in K_{g}\textrm{ and }[\bm{x}]_{\kappa}\not\in K_{g}]$. To bound this, observe that since $\mathcal{D}$ satisfies C2, instead of $K_{g}$, we can consider $K_{g}\cap B^{2}(0,n^{C})$. Since the surface area of this (convex) body is upper bounded by $n^{C(n-1)}\cdot\pi^{n-1}\cdot 2^{n}$ by Section 3.1.3, the surface area of the portion of the two hyperplanes defined by $f$ and $g$ which is within $B^{2}(0,n^{C})$ is at most $n^{C(n-1)}\cdot\pi^{n-1}\cdot 2^{n}$. We additionally recall that the density of $\mathcal{D}_{f}$ is bounded by $4^{n}$. Now, the event “$\bm{x}\in K_{g}\textrm{ and }[\bm{x}]_{\kappa}\not\in K_{g}$” implies that $\bm{x}$ lies in $B^{2}(0,n^{C})$ and is $\kappa$-close to a point on the separating hyperplane for the LTF $f$ or a point on the separating hyperplane for the LTF $g$. By the above surface area and density upper bounds, this probability can be upper bounded by

[TABLE]

(recall that by assumption the value of $\varepsilon$ is at least $1/2^{o(n)}$). Combining this with Equation 3, we conclude that $\operatorname{{\bf Pr}}_{\bm{x}\leftarrow\mathcal{D}_{f}}[[\bm{x}]_{\kappa}\in g^{-1}(-1)]\geq\varepsilon/2$. Since $g$ is a halfspace and the VC-dimension of halfspaces is $n+1$, a standard uniform convergence argument now gives the claim. ∎

Proof Sketch of New Version of Section 3.2.1.

Consider any example $(x,\ell_{x})$ fed to the online learner. We first consider the case when $\ell_{x}=1$ (corresponding to Step 5). In this case, note that $x=[\bm{y}]_{\kappa}$ and $\ell_{x}=f(\bm{y})=1$ where $\bm{y}\leftarrow\mathcal{D}_{f}$. Similar to before, note that the only way $\ell_{x}$ can differ from $f(x)$ is if $\bm{y}\in B^{2}(0,N^{c})$ and $\bm{y}$ is $\kappa$-close to the boundary of the set $K$, where now $K=f^{-1}{(1)}$. As in the argument in the previous proof sketch, this probability can be bounded by

[TABLE]

Next we consider the case when $\ell_{x}=-1$ (corresponding to Step 8). In this case, we note that the point $\bm{x}$ generated in Step 8 is indeed in $f^{-1}(-1)$ with probability at least $1-\gamma$. Consequently, $\operatorname{{\bf Pr}}[[\bm{x}]_{\kappa}\not\in f^{-1}(-1)]\leq\gamma+2^{-n}$. Since the total number of points the algorithm generates in Step 8 is at most $M$, the claim follows from the fact that $M(\gamma+2^{-n})\ll 1$ (as in the analysis of [DDS15a]). ∎

Given the new versions of Sections 3.2.1 and 3.2.1, an argument entirely analogous to the argument employed in [DDS15a] to establish (a) and (b) from Sections 3.2.1 and 3.2.1 gives us the following analogues of (a) and (b):

(a′)**

Agreement in our setting: $\operatorname{{\bf Pr}}_{\bm{x}\leftarrow\mathcal{D}_{f}}\left[\,g([\bm{x}]_{\kappa})=1\,\right]\geq 1-2\varepsilon$,

(b′)**

Density in our setting: $\operatorname{{\bf Pr}}_{\bm{x}\leftarrow\mathcal{D}}\left[\,g([\bm{x}]_{\kappa})=1\,\right]\leq(1/\gamma)\operatorname{{\bf Pr}}_{\bm{x}\leftarrow\mathcal{D}}\left[\,f(\bm{x})=1\,\right]$.

Similar arguments to those employed above show that the probability (under either ${\cal D}_{f}$ or ${\cal D}$) that $\bm{x}$ is so close to the boundary of $g$ as to have $g([\bm{x}]_{\kappa})\neq g(\bm{x})$ is extremely small. Hence conditions (a) and (b) can be inferred from (a′)** and (b′)**, as required by the definition of a densifier. This concludes the proof sketch of Theorem 23.

4 A distribution learning algorithm for degree-2 PTFs under the normal background distribution

In this section we fix the ambient distribution to be a standard Gaussian, $\mathcal{N}:=\mathcal{N}(0,1)^{n}$. We wish to learn the distribution ${\cal N}_{f}$ induced by $\mathcal{N}$ and an (unknown) degree-2 PTF $f$. In this section, we give an algorithm with time and sample complexity $\mathrm{poly}(n,1/\varepsilon)$ which learns ${\cal N}_{f}$ to total variation $\varepsilon$, given independent samples from it.

Theorem 24** (Theorem 6, restated).**

There is an algorithm $A$ with the following property: Let $f\in\mathsf{PTF}^{n}_{2}$ be such that $\operatorname{{\bf Pr}}_{\bm{x}\leftarrow{\cal N}(0,1)^{n}}[f(\bm{x})=1]\geq 2^{-n}.$ Given access to i.i.d. samples from $\mathcal{N}_{f}=$ ($\mathcal{N}(0,1)^{n}$ restricted to $f^{-1}(1)$), for $\varepsilon=1/2^{o(n)}$ algorithm $A$ runs in $\mathrm{poly}(n,1/\varepsilon)$ time and with high probability outputs a hypothesis distribution $\mathcal{D}^{\prime}$ such that $d_{\mathrm{TV}}(\mathcal{D}^{\prime},\mathcal{N}_{f})\leq\varepsilon.$

As in the previous section, to prove this theorem we will rely on the general algorithmic result of Theorem 12, with $\mathcal{C}$ and $\mathcal{C}^{\prime}$ both set to be $\mathsf{PTF}^{n}_{2}$ and $\mathcal{D}$ being $\mathcal{N}(0,1)^{n}$. That is, we will provide (i) a densifier (Section 4.1); (ii) an approximate counting algorithm (Section 4.2); (iii) a weak approximate generation algorithm (Section 4.3); and (iv) an SQ learning algorithm for $\mathsf{PTF}^{n}_{2}$.888We note that for (ii) and (iii), we will actually provide a marginally less general result, which we will show to be sufficient for our purposes given the assumptions of Theorem 24: namely, counting and generation algorithms only for $f\in\mathsf{PTF}^{n}_{2}$ such that $\operatorname{{\bf Pr}}_{\bm{x}\leftarrow{\cal N}(0,1)^{n}}[f(\bm{x})=1]\geq 2^{-n}$. In fact, (iv) follows from Theorem 21, since upon reparameterization a degree-2 PTF over $\mathbb{R}^{n}$ is a degree-1 PTF over $\mathbb{R}^{m}$, for $m:=n+\binom{n}{2}$.

4.1 Construction of densifier for $\mathsf{PTF}^{n}_{2}$ under normal distribution

In this subsection, we establish the following theorem, which provides a densifier for $\mathsf{PTF}^{n}_{2}$ under the ambient normal distribution:

Theorem 25**.**

There is an algorithm $\mathcal{A}^{\sf PTF}_{\mathsf{den}}$ which is an $(\varepsilon,\gamma,\delta)$-densifier for $\mathsf{PTF}^{n}_{2}$ using $\mathsf{PTF}^{n}_{2}$ under $\mathcal{N}$ where $\gamma=1/\mathsf{poly}(n).$ Algorithm $\mathcal{A}^{\sf PTF}_{\mathsf{den}}$ has running time $\mathsf{poly}(n,1/\varepsilon,\log(1/\delta))$.

Proof sketch.

We follow the same outline as for the proof sketch of Theorem 23. Indeed, the algorithm itself is, up to some change in parameters, the same: we will again rely on the online-mistake bound learning algorithm of Maass–Turan from Theorem 22 (observe that the Maass–Turan algorithm can be used to learn degree-2 PTFs over a discrete domain $[L]^{n}$, by viewing such a PTF as an LTF over $O(n^{2})$ variables), and we will use the same approach as in the previous section.

To make this approach work for degree-2 PTFs under the normal distribution, it suffices to (1) adapt the algorithm in two places (namely, by using the approximate counting and approximate generation algorithms for degree-2 PTFs, given in Sections 4.2 and 4.3, instead of their LTF counterparts); and (2) establish the analogues of Claims 3.2.1 and 3.2.1 in this new setting (which will correspond to slightly different choices for the parameters $N_{+}$ and $M$).

The main hurdle in doing this is that the proofs of Claims 3.2.1 and 3.2.1, in order to establish that the discretization of the space and rounding of the points (required to apply the Maass–Turan algorithm over a discrete grid of points) did not affect correctness, used the fact that the relevant sets (e.g., $g^{-1}(-1)\cap f^{-1}(1)$ and $f^{-1}(1)$) were convex sets with a very simple structure (the satisfying assignments of a single LTF or an intersection of two LTFs). This simple structure made it possible to bound the volume of the set of points in $B^{2}(0,n^{C})$ that are very close to the boundary of these sets using Fact 3.1.3. While the rest of the argument goes through for degree-2 PTFs (using, for the uniform convergence in the analogue of Claim 3.2.1, that the VC dimension of the class of degree-$2$ PTFs over $n$ real variables is $\binom{n+2}{2}$ [Ant95]), we cannot bound the probability that the rounding affects the value of the function in the same way as before since we are no longer dealing with LTFs; indeed, the relevant sets of satisfying assignments need no longer be convex.

Instead, we bound this probability more directly by exploiting the fact that the ambient distribution is Gaussian, as follows. First, we may assume without loss of generality that the current hypothesis degree-2 PTF $g=\mathrm{sign}(p)$ has been normalized, so that the squared non-constant coefficients of $p$ sum to 1. Given any $x\in B^{2}(0,n^{C})$, consider its rounding $[x]_{\kappa}$. Using Cauchy–Schwarz, the triangle inequality, and the fact that the coefficients of $p$ have been normalized as described above, it is not hard to see that

[TABLE]

Consequently, one can only have $g(x)\neq g([x]_{\kappa})$ if $p(x)$ itself is very small, namely at most $O(\kappa n^{C})$. However, invoking a well-known result of Carbery and Wright [CW01], this itself only can happen with probability $O(\sqrt{\tau})=O(\sqrt{\kappa n^{C}})$ under the standard normal distribution:

Theorem 26** ([CW01]).**

Let $p\colon\mathbb{R}^{n}\to\mathbb{R}$ be a non-identically zero degree-$d$ polynomial. Then, for all $\varepsilon>0$ and $\theta\in\mathbb{R}$,

[TABLE]

Since $p$ is degree-2, $\operatorname{{\bf Var}}[p]$ is the sum of the squared degree-1 and degree-2 Hermite coefficients when $p$ is expressed in the Hermite basis of polynomials that are orthonormal under ${\cal N}(0,1)^{n}$. Recalling that the degree-1 univariate Hermite polynomial is $h_{1}(x)=x$ and the degree-2 univariate Hermite polynomial is $h_{2}(x)={\frac{x^{2}-1}{\sqrt{2}}}$, the fact that $p$’s squared non-constant coefficients sum to 1 implies that $\operatorname{{\bf Var}}[p]=\Theta(1).$ In our setting $\theta=0$, and $d=2$, and so applying Theorem 26 we get the claimed bound.

We can use this to derive the counterparts of the main probability bounds from Claim 3.2.1 and Claim 3.2.1, respectively, allowing the proofs of the two analogous statements to go through. First, letting $K_{g}:=g^{-1}(1)\cap f^{-1}(-1)$, we have

[TABLE]

recalling that $\operatorname{{\bf Pr}}_{\bm{x}\leftarrow\mathcal{N}}\left[\,f(\bm{x})=1\,\right]\geq 2^{-n}$ and the setting of $\kappa$. This (along with the aforementioned VC dimension argument) yields the counterpart of Claim 3.2.1. A similar argument applied to the degree-2 PTF $f$ (instead of $g$) allows us, considering $K:=f^{-1}(-1)$ and following the outline of the proof of Claim 3.2.1, to establish the counterpart of that second claim as well. ∎

4.2 Approximate counting: Gaussian integration under quadratic constraints

Recall that our goal in this subsection is an algorithm with the following property: given the full description of a degree-2 PTF $g=\mathrm{sign}(p)$ over $\mathbb{R}^{n}$ such that $g^{-1}(1)$ has mass at least $1/2^{n}$ under $\mathcal{N}$, the algorithm should efficiently output a (multiplicatively accurate) estimate of $\mathop{{\bf Pr}\/}_{\bm{G}\sim\mathcal{N}}[g(\bm{G})=1].$ The main theorem of this subsection is the following.

Theorem 27**.**

There is a deterministic algorithm $\mathcal{AC}$ with the following guarantee: Given as input a degree-2 polynomial $p$ over $\mathbb{R}^{n}$ such that $g^{-1}(1)$ has mass at least $1/2^{n}$ under $\mathcal{N}$ (where $g=\mathrm{sign}(p)$) and input $\varepsilon\in(0,1]$ such that $\varepsilon\geq 1/2^{O(n)}$, $\mathcal{AC}$ runs in time $\mathrm{poly}(n,1/\varepsilon)$ and outputs an $(1+\varepsilon)$-approximation to $\operatorname{{\bf Pr}}_{\bm{G}\leftarrow\mathcal{N}}\left[\,g(\bm{G})=1\,\right]$.

Although sampling $\bm{G}=(\bm{G}_{1},\dots,\bm{G}_{n})\leftarrow\mathcal{N}$ is easy, the additional quadratic constraint significantly complicates the problem of both (i) estimating $\operatorname{{\bf Pr}}\!\left[\,p(\bm{G})\geq 0\,\right]$ as well as (ii) sampling $\bm{G}$ conditioned on $\operatorname{{\bf Pr}}\!\left[\,p(\bm{G})\geq 0\,\right]$. Indeed, as is the case with many sampling and approximate counting problems [JVV86], the complexities of (i) and (ii) are closely related. We note that these problems are of interest in the statistics literature [PP14], but rigorous guarantees were previously not known.

To prove Theorem 27, we leverage an approximate counting algorithm due to Li and Shi [LS14] for the knapsack problem (which in turn builds on the algorithm of [ŠVV12]). To state the result of Li and Shi, we first need the definition of an oracle to a probability distribution.

Definition 28**.**

For any real valued random variable $\bm{X}$, an oracle $\mathcal{O}_{\bm{X}}$ takes inputs $\nu_{1}\leq\nu_{2}\in\mathbb{R}$ and outputs the quantity $\operatorname{{\bf Pr}}[\nu_{1}\leq\bm{X}\leq\nu_{2}]$.

With this, we are ready to state the result of Li and Shi.

Theorem 29** ([LS14, Theorem 1.1]).**

There exists an algorithm $\mathcal{LS}$ with the following guarantees. Given as input an integer $C>0$, an approximation parameter $\varepsilon>0$, and oracle access to $n$ independent integer-valued random variables $\bm{X}_{1},\dots,\bm{X}_{n}$ such that $0\in\operatorname{supp}(\bm{X}_{i})\subseteq\{0,1,\dots,C+1\}$ for all $1\leq i\leq n$, the algorithm outputs a value $\widehat{\rho}$ such that

[TABLE]

Moreover, $\mathcal{LS}$ runs in time $\mathrm{poly}(n,\frac{1}{\varepsilon},\log C,\log\frac{1}{\Delta})$, where $\Delta:=\prod_{i=1}^{n}\operatorname{{\bf Pr}}\!\left[\,\bm{X}_{i}=0\,\right]$.

To see why Theorem 29 is useful, we begin by observing that, upon rotating $\mathbb{R}^{n}$ and translating the Gaussians, the approximate counting problem considered in Theorem 27 is equivalent to the following problem:

Given $(\lambda_{i},\mu_{i})_{i\in[n]}$, $\varepsilon\in(0,1]$, and $\theta\in\mathbb{R}$ such that (4) is at least $2^{-n}$, output an $(1\pm\varepsilon)$-approximation of

[TABLE]

Since both (4) and Theorem 29 deal with the probability that a sum of independent random variables exceeds a threshold, the relevance of Theorem 29 should now be clear.

In order to prove Theorem 27 using Theorem 29 using the formulation (4), we will need to do two preprocessing steps. These steps will be useful in Section 4.3 as well.

Pre-Processing Steps: Discretization and Rounding:

Let us set $\gamma:=1/2^{n^{C_{1}}}$ and $\tau:=1/2^{n^{C_{1}}}$ for suitably big constants $C_{1},C_{2}>0$. First, without loss of generality, we may assume that the coefficients in (4) satisfy $\sum_{i=1}^{n}(\lambda_{i}^{2}+\mu_{i}^{2})=1$. We can now round the coefficients $\{\lambda_{i}\}_{i=1}^{n}$ and $\{\mu_{i}\}_{i=1}^{n}$ to the nearest integral multiple of $\gamma$; we call the resulting rounded coefficients $\lambda^{\prime}_{i}$ and $\mu^{\prime}_{i}$. We observe that $\sum_{i=1}^{n}((\lambda_{i}-\lambda^{\prime}_{i})^{2}+(\mu_{i}-\mu^{\prime}_{i})^{2})\leq O(n\gamma^{2})$, and that we also have

[TABLE]

We now recall the following useful fact from [DS17]:

Lemma 4.1** ([DS17, Lemma 6]).**

Let $a(x)$ and $b(x)$ be multivariate degree-$d$ polynomials over $\mathbb{R}^{n}$ such that for $\mathbf{G}\leftarrow\mathcal{N}$, $\mathbb{E}\!\left[a(\bm{G})-b(\bm{G})\right]=0$, $\operatorname{{\bf Var}}[a]=1$ and $\operatorname{{\bf Var}}[a-b]\leq(\tau/d)^{3d}$. Then,

[TABLE]

Recalling that $p(x)=\sum_{i=1}^{n}(\lambda_{i}x_{i}^{2}+\mu_{i}x_{i})-\theta$, we define $q(x)=\sum_{i=1}^{n}(\lambda^{\prime}_{i}x_{i}^{2}+\mu^{\prime}_{i}x_{i})-\theta$. Applying Lemma 4.1 to $p(x)$ and $q(x)$ with $d=2$ and $\tau=O(n^{\frac{1}{6}}\cdot\gamma^{\frac{1}{3}})$, we get that

[TABLE]

For a suitable choice of the constant $C_{1}$, the right hand side is at most $O(2^{-n^{2}})$. Since $\operatorname{{\bf Pr}}[\mathrm{sign}(p(\bm{G}))=1]\geq 2^{-O(n)}$, this implies the following claim:

Claim \theclaim.

Let $g=\mathrm{sign}(p)$ and $h=\mathrm{sign}(q)$. For any $\varepsilon\geq 2^{-O(n)}$, $\operatorname{{\bf Pr}}_{\bm{G}\leftarrow\mathcal{N}}[g(\bm{G})=1]$ is an $(1+\varepsilon)$-approximation of $\operatorname{{\bf Pr}}_{\bm{G}\leftarrow\mathcal{N}}[h(\bm{G})=1]$. Similarly, $\|\bm{G}_{h}-\bm{G}_{g}\|_{1}\leq 2^{-O(n)}$. Thus, to sample from $\bm{G}_{g}$, it suffices to design an efficient sampling algorithm for $\bm{G}_{h}$.

We next define a discretization $[\bm{G}]_{\tau}:=([\bm{G}_{1}]_{\tau},\dots,[\bm{G}_{n}]_{\tau})$ of the Gaussian random variable $\bm{G}=(\bm{G}_{1},\ldots,\bm{G}_{n})$ as follows: each $[\bm{G}_{i}]_{\tau}$ is only supported on the points $\mathsf{AP}_{-n,n,\tau}:=\{-n,-n+\tau,\ldots,n-\tau,n\}$. For each point $\kappa\in\mathsf{AP}_{-n,n,\tau}$,

[TABLE]

In other words, we assign each point $\kappa\in\mathsf{AP}_{-n,n,\tau}$, the probability mass that the standard Gaussian puts in the interval $[\kappa,\kappa+\tau)$ (with the exception of $-n$ and $n$, whose probability is slightly increased by the capping).

To proceed further, for conciseness let us adopt the shorthand $\bm{H}=(\bm{H}_{1},\ldots,\bm{H}_{n})$ to denote the random variable $[\bm{G}]_{\tau}=([\bm{G}_{1}]_{\tau},\dots,[\bm{G}_{n}]_{\tau})$. We next have the following claim.

Claim \theclaim.

For $q(x)$ as specified above, we have that

[TABLE]

Proof.

First of all, note that $\operatorname{{\bf Pr}}[|\bm{G}_{i}|>n]\leq 2^{-\Theta(n^{2})}$ for each $i\in[n]$. Let $\mathbf{E}$ be the event that for all $1\leq i\leq n$, $|\bm{G}_{i}|\leq n$. By a union bound, we obtain

[TABLE]

Using the upper bound on coefficients of $q$ from (5), we have that conditioned on event $\mathbf{E}$, $|q(\bm{G})-q(\bm{H})|\leq O(n^{2}\tau)$. Consequently, conditioned on event $\mathbf{E}$, $\mathrm{sign}(q(\bm{G}))\not=\mathrm{sign}(q(\bm{H}))$ only if $|q(\bm{G})|=O(n^{2}\cdot\tau)$. However, by anti-concentration of quadratic polynomials (Theorem 26), we have

[TABLE]

The above application of Theorem 26 uses the lower bound on the sum of squares of the coefficients of $q$ from (5). Combining (8) and (7) yields the claim. ∎

In light of Sections 4.2 and 4.2, it suffices to give an algorithm to compute $\operatorname{{\bf Pr}}_{\bm{H}}[\mathrm{sign}(q(\bm{H}))=1]$ where $q$ and $\bm{H}$ are as defined above; we do this below.

Approximate counting of $\operatorname{{\bf Pr}}_{\bm{H}}[\mathrm{sign}(q(\bm{H}))=1]$.

Recall that $q(\bm{H})=\sum_{i=1}^{n}({\lambda^{\prime}}_{i}^{2}\bm{H}_{i}^{2}+{\mu^{\prime}}_{i}\bm{H}_{i})-\theta$ and that our plan is to leverage the algorithm of [LS14] (Theorem 29) to approximate $\operatorname{{\bf Pr}}_{\bm{H}}[\mathrm{sign}(q(\bm{H}))=1]$. Our argument employs the following three claims.

Claim \theclaim.

Define the random variable $\bm{X}_{i}=\lambda^{\prime}_{i}\bm{H}_{i}^{2}+\mu^{\prime}_{i}\bm{H}_{i}$. This random variable is supported on integral multiples of $\gamma\tau^{2}=2^{-n^{C_{1}}-2n^{C_{2}}}$ bounded in the range $[-2n^{2},2n^{2}]$.

Proof.

The range of this random variable can be bounded just by observing that each $\lambda^{\prime}_{i}$ and $\mu^{\prime}_{i}$ is bounded in absolute value by $1$. Further, since $\lambda^{\prime}_{i}$ and $\mu^{\prime}_{i}$ are integral multiples of $\gamma$ and $\bm{H}$ is supported on integral multiples of $\tau$, the random variable $\bm{X}_{i}$ is supported on integral multiples of $\gamma\tau^{2}$. ∎

Claim \theclaim.

Let $\alpha$ be such that $\operatorname{{\bf Pr}}[\bm{X}_{i}=\alpha]>0$. Then, $\operatorname{{\bf Pr}}[\bm{X}_{i}=\alpha]\geq 2^{-\Theta(n^{C_{1}})}$.

Proof.

Note that any draw of $\bm{X}_{i}$ is completely determined by the corresponding draw of $\bm{H}_{i}$. Thus, it suffices to show that

[TABLE]

Note that at any point $x\in[-n,n]$, the density of $\bm{G}_{i}$ is at least $2^{-\Theta(n^{2})}$. Note further that any support point of $\bm{H}_{i}$ lies in the set $\mathsf{AP}_{-n,n,\tau}$. For any point $\alpha\neq n$ in this set, one can easily lower bound $\bm{H}_{i}(\alpha)$ as

[TABLE]

For $\alpha=n$,

[TABLE]

So, for every possibility for $\alpha$, we get the claimed lower bound on $\bm{H}_{i}(\alpha)$, finishing the proof. ∎

Claim \theclaim.

Let $a,b\in\mathbb{R}$ where $|a|,|b|\leq 1$. Let $\bm{Y}$ denote the random variable defined by $a{[\bm{G}]_{\tau}}^{2}+{b}[\bm{G}]_{\tau}$ where $\bm{G}$ is a standard normal random variable. Assuming $a$ and $b$ are rational multiples of $2^{-\mathrm{poly}(n)}$, we can implement $\mathcal{O}_{\bm{Y}}$ in $\mathrm{poly}(n)$ time.

Proof.

First, we observe that given any $a$ and $b$ as above, the set

[TABLE]

is given by a union of at most two closed intervals $I_{1}$ and $I_{2}$ where the boundary points of $I_{1}$ and $I_{2}$ can be (efficiently) obtained by solving quadratic equations in one-variable. Moreover, in view of the definition of $[\bm{G}]_{\tau}$ from $\bm{G}$, for any interval $I$ we have that $\operatorname{{\bf Pr}}[[\bm{G}]_{\tau}\in I]=\operatorname{{\bf Pr}}[\bm{G}\in J]$, where $J\supseteq I$ is an efficiently computable interval (depending only on the endpoints of $I$ and on $\tau$). Finally, given any interval $J$, the value $\operatorname{{\bf Pr}}[\bm{G}\in J]$ can clearly be computed in polynomial time. This finishes the claim. ∎

Given the above three claims, we are left in a position where we can apply Theorem 29 to the discrete random variables

[TABLE]

where $m_{i}=\min_{-n\leq j\leq n}(\lambda^{\prime}_{i}j^{2}+\mu^{\prime}_{i}j)$ (this ensures that indeed [math] belongs to the support of $\bm{X}^{\prime}_{i}$). By Section 4.2, the $\bm{X}^{\prime}_{i}$’s are thus non-negative integer-valued random variables with $0\in\operatorname{supp}(\bm{X}^{\prime}_{i})\subseteq\{0,1,\dots,\frac{|m_{i}|+2n^{2}}{\gamma\tau^{2}}\}\subseteq\{0,1,\dots,\frac{4n^{2}}{\gamma\tau^{2}}\}$. The parameter $C$ in Theorem 29 is therefore upper bounded by $\frac{4n^{2}}{\gamma\tau^{2}}=2^{n^{C_{1}}+2n^{C_{2}}+o(n)}$, so $\log C$ is indeed $\mathrm{poly}(n)$. Moreover, we can implement the required oracles $\{\mathcal{O}_{\bm{X}^{\prime}_{i}}\}_{i=1}^{n}$ with a $\mathrm{poly}(n)$-time overhead by Section 4.2.

To conclude and be able to claim a $\mathrm{poly}(n,1/\varepsilon)$ runtime overall after invoking Theorem 29, we also require a $2^{-\mathrm{poly}(n)}$ bound on the parameter $\Delta=\prod_{i=1}^{n}\operatorname{{\bf Pr}}\!\left[\,\bm{X}^{\prime}_{i}=0\,\right]$. This bound follows from Section 4.2: for any $\alpha\in\operatorname{supp}(\bm{X}^{\prime}_{1})\times\dots\times\operatorname{supp}(\bm{X}^{\prime}_{n})$, we have $\prod_{i=1}^{n}\operatorname{{\bf Pr}}\!\left[\,\bm{X}^{\prime}_{i}=\alpha_{i}\,\right]\geq\left(2^{-\Theta(n^{C_{1}})}\right)^{n}=2^{-\Theta(n^{C_{1}+1})}$. Since $0\in\operatorname{supp}(\bm{X}^{\prime}_{i})$ for all $i$, this leads to the same lower bound on $\Delta$, and applying Theorem 29 concludes the proof of Theorem 27.

4.3 Weak approximate generation: Gaussian sampling under quadratic constraints

In this subsection, we will prove the following theorem.

Theorem 30**.**

There is an efficient algorithm $\mathcal{RS}$ with the following guarantee: Given as input a degree-2 polynomial $p$ defining a PTF $g=\mathrm{sign}(p)$ over $\mathbb{R}^{n}$ such that $g^{-1}(1)$ has mass at least $2^{-n}$ under $\mathcal{N}$, and input $\varepsilon\in(0,1]$ such that $\varepsilon\geq 1/2^{O(n)}$, the algorithm runs in time $\mathrm{poly}(n/\varepsilon)$ and outputs a point $x$ distributed according to a distribution $\mathcal{D}$ such that $\|\mathcal{D}-\bm{G}_{g}\|_{1}\leq\varepsilon$.

Similar to Section 4.2, we can assume that $p$ is of the form

[TABLE]

As in the previous section, we set parameters $\gamma:=1/2^{n^{C_{1}}}$ and $\tau:=1/2^{n^{C_{1}}}$ for suitably big constants $C_{1},C_{2}>0$, we define the coefficients $\lambda^{\prime}_{i}$ and $\mu^{\prime}_{i}$ obtained by rounding the parameters $\lambda_{i}$ and $\mu_{i}$ to the nearest integral multiple of $\gamma$ (identical to Section 4.2), and we define $q(x)=\sum_{i=1}^{n}\lambda^{\prime}_{i}x_{i}^{2}+\sum_{i=1}^{n}\mu^{\prime}_{i}x_{i}-\theta$ and $h=\mathrm{sign}(q)$. Using Claim 4.2, it suffices to give an efficient sampling algorithm for the distribution $\bm{G}_{h}$.

We define the random variable $\bm{H}$ as in (6). We also define $\bm{G}_{h(\bm{H})}$ as the random variable $\bm{G}$ conditioned on $h(\bm{H})=1$. Finally, $\bm{H}_{h}$ denotes the distribution $\bm{H}$ conditioned on $h(\bm{H})=1$.

By Claim 4.2, we have that $\|\bm{G}_{h}-\bm{G}_{h(\bm{H})}\|_{1}\leq 2^{-\Theta(n^{2})}$. Thus, it suffices to produce a sampler for the distribution $\bm{G}_{h(\bm{H})}$. Next, observe that since our algorithms are efficient, it suffices to sample points $\bm{G}_{h(\bm{H})}$ up to $n^{\Theta(1)}$ bits of precision. We further recall that $\operatorname{{\bf Pr}}[\|\bm{G}\|_{\infty}>n]\leq 2^{-\Theta(n^{2})}$. Together, these observations imply that it suffices to sample from $\bm{H}_{h}$. To accomplish this, we use the generic idea of reducing sampling to approximate counting [JVV86]. We begin with the following definition.

Definition 31**.**

Let $\overline{\kappa}^{(1)},\ \overline{\kappa}^{(2)}\in\mathsf{AP}_{-n,n,\tau}^{n}$ where $\overline{\kappa}^{(1)}_{j}\leq\overline{\kappa}^{(2)}_{j}$ for every $1\leq j\leq n$ (this is denoted by $\overline{\kappa}^{(1)}\preceq\overline{\kappa}^{(2)}$). Then the random variable $\bm{H}_{\overline{\kappa}^{(1)},\overline{\kappa}^{(2)}}$ represents $\bm{H}$ conditioned on the $j^{th}$ coordinate lying in the interval $[\overline{\kappa}^{(1)}_{j},\overline{\kappa}^{(2)}_{j}]$ for every $1\leq j\leq n.$

We now claim that the algorithm from the preceding subsection can be augmented to achieve the following guarantee:

Theorem 32**.**

There exists a deterministic algorithm $\mathcal{AC}^{\prime}$ with the following guarantee: Given a degree-$2$ polynomial $q:\mathbb{R}^{n}\rightarrow\mathbb{R}$ (satisfying the above conditions), error parameter $\varepsilon\geq 2^{-O(n)}$, and vectors $\overline{\kappa}^{(1)}\preceq\ \overline{\kappa}^{(2)}\in\mathsf{AP}_{-n,n,\tau}^{n}$, algorithm $\mathcal{AC}^{\prime}$ returns a $(1\pm\varepsilon)$ approximation to the quantity $\operatorname{{\bf Pr}}[h(\bm{H}_{\overline{\kappa}^{(1)},\overline{\kappa}^{(2)}})=1]$.

Briefly, the algorithm in the preceding subsection relies on three crucial claims:

The random variable $\bm{X}_{i}=\lambda^{\prime}_{i}\bm{H}_{i}^{2}+\mu^{\prime}_{i}\bm{H}_{i}$ is an integral multiple of $\gamma\tau^{2}=2^{-n^{C_{1}}-2n^{C_{2}}}$ bounded in the range $[-2n^{2},2n^{2}]$ (Claim 4.2). Here $\bm{H}_{i}$ is the $i^{th}$ coordinate of $\bm{H}$. Let us now define $\bm{X}_{i,\overline{\kappa}^{(1)},\overline{\kappa}^{(2)}}$ by replacing $\bm{H}$ with $\bm{H}_{\overline{\kappa}^{(1)},\overline{\kappa}^{(2)}}$ in the definition of $\bm{X}_{i}$. Note that the support of $\bm{H}_{\overline{\kappa}^{(1)},\overline{\kappa}^{(2)}}$ is a subset of the support of $\bm{H}$. Thus, the support of $\bm{X}_{i,\overline{\kappa}^{(1)},\overline{\kappa}^{(2)}}$ is a subset of the support of $\bm{X}_{i}$, and as a result, Claim 4.2 will continue to hold true for $\bm{X}_{i,\overline{\kappa}^{(1)},\overline{\kappa}^{(2)}}$. 2. 2.

For any $\alpha$ such that $\operatorname{{\bf Pr}}[\bm{X}_{i}=\alpha]>0$, we have that $\operatorname{{\bf Pr}}[\bm{X}_{i}=\alpha]\geq 2^{-\Theta(n^{C_{1}})}$ (Claim 4.2). It is easy to see that the same proof also implies Claim 4.2 when $\bm{X}_{i}$ is replaced by $\bm{X}_{i,\overline{\kappa}^{(1)},\overline{\kappa}^{(2)}}$. 3. 3.

Let $a,b$ be rational multiples of $2^{-\mathrm{poly}(n)}$ of magnitude at most 1. Let $\bm{Y}$ denote the random variable $a\bm{H}_{i}^{2}+b\bm{H}_{i}$. Then, we can implement the oracle $\mathcal{O}_{\bm{Y}}$ in polynomial time (Claim 4.2). It is easy to see that this continues to hold if we replace $\bm{H}_{i}$ by $\bm{H}_{i,\overline{\kappa}^{(1)},\overline{\kappa}^{(2)}}$.

As the analogues of Claim 4.2, Claim 4.2 and Claim 4.2 hold for $\bm{X}_{i,\overline{\kappa}^{(1)},\overline{\kappa}^{(2)}}$, the proof of correctness of the algorithm in the previous subsection can be modified mutatis mutandis to prove Theorem 32.

We now complete the proof of Theorem 30 by presenting and analyzing the algorithm $\mathcal{RS}.$ The algorithm is a recursive routine and is given below; it is a straightforward translation of the usual counting-to-sampling reduction to our setting (augmented with binary search since in our context we are dealing with larger domains for each coordinate than $\{0,1\}$).

1. (1)

Initialize $\overline{\kappa}^{\ell}=(-n,-n,\ldots,-n)$ and $\overline{\kappa}^{u}=(n,n,\ldots,n)$. 2. (2)

Initialize $\mathcal{A}$ as $\mathcal{A}_{\varepsilon}=[\overline{\kappa}^{\ell}_{1},\overline{\kappa}^{u}_{1}]\times\ldots\times[\overline{\kappa}^{\ell}_{n},\overline{\kappa}^{u}_{n}]\cap(\mathsf{AP}_{-n,n,\tau})^{n}$. 3. (3)

Set $\delta=\varepsilon/(2\log|\mathcal{A}|)$. 4. (4)

If $|\mathcal{A}|=1$, then return $\mathcal{A}$. 5. (5)

Else, choose the first $j$ such that $[\overline{\kappa}^{\ell}_{j},\overline{\kappa}^{u}_{j}]\cap\mathsf{AP}_{-n,n,\tau}>1$. Let $\eta^{\textsf{mid}}=(\overline{\kappa}^{j}_{\ell}+\overline{\kappa}^{j}_{u})/2$. Define $\overline{\kappa}^{1,\textsf{mid}}$ and $\overline{\kappa}^{2,\textsf{mid}}$ as

[TABLE]

[TABLE] 6. (6)

Invoke Theorem 32 to obtain $(1\pm\delta)$ approximations to the quantities $\operatorname{{\bf Pr}}[h(\bm{H}_{\overline{\kappa}^{1},\overline{\kappa}^{1,\textsf{mid}}})=1]$ and $\operatorname{{\bf Pr}}[h(\bm{H}_{\overline{\kappa}^{2,\textsf{mid}},\overline{\kappa}^{2}})=1]$. 7. (7)

Call these approximations $\eta_{1}$ and $\eta_{2}$. Let $\mathbf{b}$ be a Bernoulli random variable where $\operatorname{{\bf Pr}}[\mathbf{b}=0]=\eta_{1}/(\eta_{1}+\eta_{2})$. 8. (8)

Define $\mathcal{A}_{0}$ and $\mathcal{A}_{1}$ as

[TABLE]

[TABLE] 9. (9)

Sample $b\leftarrow\mathbf{b}$ and go to Step 2 with $\mathcal{A}\leftarrow\mathcal{A}_{b}$.

We now make the following claim.

Claim \theclaim.

The above procedure terminates after $\log|\mathcal{A}|$ repetitions. The total running time is $\mathrm{poly}(n,1/\varepsilon)$.

Proof.

Note that at each stage $\mathcal{A}$ shrinks in size by a factor of $2$ and when $|\mathcal{A}|=1$, then the procedure terminates. This proves the first part of the claim. The second part follows from the fact that in each iteration every step (including Step 6) runs in time $\mathrm{poly}(n,1/\varepsilon)$. ∎

Next, we observe that corresponding to any string $z$ of length at most $m=\lceil\log|\mathcal{A}|\rceil$, we can associate a set $\mathcal{A}_{z}$ as follows:

Set $\mathcal{A}\leftarrow\mathcal{A}_{\varepsilon}$ (recall step (2) of algorithm $\mathcal{RS}$). 2. 2.

Set $j\leftarrow 1$. 3. 3.

If $|\mathcal{A}|=1$ or $j=|z|+1$, return $\mathcal{A}_{z}:=\mathcal{A}$. 4. 4.

Else, we define $\mathcal{A}_{0}$ and $\mathcal{A}_{1}$ by Steps (5) and (8) of the above algorithm. 5. 5.

Let $b$ be the $j^{th}$ bit of $z$. Then, set $\mathcal{A}\leftarrow\mathcal{A}_{b}$ and $j\leftarrow j+1$. Go to Step 3.

We observe that (1) the above routine associates a unique string of length at most $m$ with any element $x\in\mathcal{A}_{\varepsilon}$. (2) If a string $z$ is a prefix of a string $w$, then $\mathcal{A}_{w}\subseteq\mathcal{A}_{z}$.

We are now ready for the following claim:

Claim \theclaim.

Let $z$ be any string and let $\mathcal{A}_{z}\subseteq\mathcal{A}_{\varepsilon}$ defined above. Define $p_{\mathsf{sample},z}$ to be the probability that $\mathcal{RS}$ samples an element from $\mathcal{A}_{z}$. Let $p_{\mathsf{true},z}=\operatorname{{\bf Pr}}_{\bm{H}}[\bm{H}\in\mathcal{A}_{z}|h(\bm{H})=1].$ Then

[TABLE]

Proof.

The proof is a simple induction on the length of $z$. First, observe that the bounds trivially hold when $z=\varepsilon$. Now, inductively assume that the bounds hold when the length of $z$ is at most $\ell$. Let us consider any $z=z^{\prime}\circ b$ where $b\in\{0,1\}$. For steps (5), (6) and (7) of the above algorithm, define $\eta_{1,\mathsf{true}}$ to be $\operatorname{{\bf Pr}}[h(\bm{H}_{\overline{\kappa}^{1},\overline{\kappa}^{1,\textsf{mid}}})=1]$ and $\eta_{2,\mathsf{true}}$ to be $\operatorname{{\bf Pr}}[h(\bm{H}_{\overline{\kappa}^{2,\textsf{mid}},\overline{\kappa}^{2}})=1]$. Further, let $\eta_{1}$ and $\eta_{2}$ be the $(1\pm\delta)$ approximations to $\eta_{1}$ and $\eta_{2}$ respectively that are obtained in step (6). We have that

[TABLE]

Using the induction hypothesis for $z^{\prime}$ and the fact that $\eta_{b}$ is a $(1\pm\delta)$ approximation of $\eta_{b,\mathsf{true}}$, we get the claim for $z$. ∎

To complete the proof of Theorem 30, it remains only to recall that $\delta=\varepsilon/2m$.

5 Hardness results for learning from satisfying assignments

In this section we show that under various standard cryptographic assumptions, our algorithmic results from the previous sections are close to the strongest possible for learning with respect to log-concave and normal background distributions. In particular, we will show that under cryptographic assumptions, there is no efficient algorithm to

learn the class of $n$-variable degree-two polynomial threshold functions $\mathsf{PTF}_{2}^{n}$ when the background distribution is a known log-concave distribution. In fact, this lower bound holds even when the background distribution is the uniform distribution over the solid $n$-dimensional hypercube $[0,1]^{n}$ (which is an extremely simple log-concave distribution). 2. 2.

learn the class of $n$-variable degree-four polynomial threshold functions $\mathsf{PTF}_{4}^{n}$ when the background distribution is the $n$-dimensional standard Gaussian $\mathcal{N}(0,1)^{n}.$

We obtain these results using (an extension of) a general condition for showing hardness of learning from [DDS15a]. We begin by recalling the notion of an invertible Levin reduction.

Definition 33**.**

A binary relation $R$ is said to reduce to another binary relation $R_{0}$ by a polynomial-time invertible Levin reduction if there are three algorithms $\alpha(\cdot)$, $\beta(\cdot,\cdot)$ and $\gamma(\cdot,\cdot)$, each running in polynomial time, with the following properties:

For each $x,y$, if $(x,y)\in R$ then $(\alpha(x),\beta(x,y))\in R_{0}$. 2. 2.

For each $x,z$, if $(\alpha(x),z)\in R_{0}$ then $(x,\gamma(\alpha(x),z))\in R.$ 3. 3.

Finally, the functions $\beta$ and $\gamma$ are such that for each $x,y$, we have $\gamma(\alpha(x),\beta(x,y))=y$.

In the context of this paper, we note that for any class $\mathcal{C}$ of Boolean functions (over any domain), we can define a relation $R_{\mathcal{C}}$ which contains those pairs $(f,z)$ such that $f\in\mathcal{C}$ and $f(z)=1$. In this section, whenever we say that there is an invertible Levin reduction from class $\mathcal{C}_{1}$ to class $\mathcal{C}_{2}$, we mean that there is an invertible Levin reduction from $\mathcal{R}_{\mathcal{C}_{1}}$ to $\mathcal{R}_{\mathcal{C}_{2}}$. As an illustrative example, we may take $\mathcal{C}_{1}$ to be the class of all polynomial-size Boolean circuits (corresponding to the Circuit-SAT problem) and $\mathcal{C}_{2}$ to be the class of all 3-CNF formulas. For property (1), given a circuit $C$ and a satisfying assignment $z$, the output $\alpha(C)$ is the 3-CNF formula (over an expanded variable space) that is produced by the standard reduction, and if $(C,z)$ belongs to $\mathcal{R}_{\mathcal{C}_{1}}$ (meaning that $z$ is a satisfying assignment for circuit $C$) then the output $\beta(C,z)$ is the corresponding satisfying assignment for the 3-CNF formula $\alpha(C).$ For property (2), given a 3-CNF $\alpha(C)$ and a satisfying assignment $z^{\prime}$ of that 3CNF, $\gamma(\alpha(C),z^{\prime})$ outputs the (unique) satisfying assignment $z$ of $C$ that “gave rise to” $z^{\prime}$, and it is easy to see that property (3) holds.

Next, we recall a standard assumption from the cryptographic literature [MRV99], which is a slight variant of the standard RSA assumption.

Assumption 1**.**

Let $\mathsf{RSA}_{k}$ be the set of all integers which are products of two primes of length $\lfloor(k-1)/2\rfloor$. Let $\bm{m}$ be chosen uniformly from $\mathsf{RSA}_{k}$ and let $\bm{x}$ be chosen uniformly from $\mathbb{Z}_{\bm{m}}^{\ast}$. Let $\bm{p}$ be a uniformly chosen prime of length $k+1$. There exists some absolute constant $\delta>0$ such that, for any probabilistic algorithm $A$ running in time $2^{n^{\delta}}$,

[TABLE]

Under the above assumption [MRV99] showed the existence of a so-called unique signature scheme which is secure for subexponential time algorithms under “random message attack” (RMA). In [DDS15a], the authors showed how the existence of such unique signature schemes implies hardness of learning under the uniform background distribution over $\{-1,1\}^{n}$ for classes $\mathcal{C}$ for which there is an invertible Levin reduction from Circuit-SAT. More precisely, [DDS15a] established the following theorem.

Theorem 34**.**

Let $\mathcal{C}$ be a class of functions such that there is a polynomial-time invertible Levin reduction from Circuit-SAT to $\mathcal{C}$. Then, under 1, there exists an absolute constant $\delta^{\prime}>0$ such that no $2^{n^{\delta^{\prime}}}$-time algorithm can learn $\mathcal{C}$ under the uniform background distribution.

In the current work we will need a slight extension of Theorem 34 which is suitable for establishing hardness when the background distribution is something other than the uniform distribution on $\{-1,1\}^{n}$. The precise result we shall use is stated below.

Theorem 35**.**

Let $\mathcal{C}$ be a class of functions from $\mathbb{R}^{n}$ to $\{-1,1\}$ such that there is a polynomial-time invertible Levin reduction from Circuit-SAT to $\mathcal{C}$. Suppose further that the background distribution $\mathcal{D}$ and the reduction are such that the following properties hold:

There is an efficient algorithm which, on input a point $x$, outputs the value of the probability density function for ${\cal D}$ at $x$ (i.e., an efficient algorithm to simulate an evaluation oracle for ${\cal D}$). 2. 2.

If $C$ is a circuit which is an instance of Circuit-SAT and $f\in\mathcal{C}$ is an instance of ${\cal C}$ which arises from $C$ in the reduction, then $f^{-1}(1)$ is the disjoint union of some collection of $K$ sets $S_{1},\dots,S_{K}$, where $K$ is the number of satisfying assignments for $C$ and each set $S_{i}$ corresponds to precisely one solution to $C$. Moreover,

1. (a)

For any $i,j\leq K$, the regions $S_{i}$ and $S_{j}$ have probability mass within a factor of two of each other under ${\cal D}$; and 2. (b)

There is an efficient algorithm which, for any $j\in[K]$, given any $x\in S_{j}$, outputs a sample drawn from ${\cal D}_{S_{j}}$ (the distribution ${\cal D}$ restricted to $S_{j}$).

Then, under 1, there exists an absolute constant $\delta^{\prime}>0$ such that no $2^{n^{\delta^{\prime}}}$-time algorithm can learn $\mathcal{C}$ under the background distribution $\mathcal{D}$.

Inspection of the proof of Theorem 34 shows that the arguments used in its proof straightforwardly extend to yield Theorem 35. Intuitively, Item 1 ensures that given the hypothesis distribution generated by a learning algorithm for $\mathcal{C}$ with respect to background distribution ${\cal D}$ (recall that in our model such a hypothesis distribution requires query access to an evaluation oracle for ${\cal D}$), it is indeed possible to generate samples from a distribution that is statistically close to ${\cal D}_{f}$. Intuitively, Item 2, together with the invertible Levin reduction, makes it possible to translate a draw from the hypothesis distribution back to a signed message and thereby contradict 1.

5.1 A lower bound for degree-2 PTFs under log-concave background distributions

In this subsection we prove Theorem 2.

Theorem 36** (Theorem 2, restated).**

Under 1, there is no subexponential-time algorithm $A$ for learning the class $\mathsf{PTF}^{n}_{2}$ of $n$-variable degree-two polynomial threshold functions with respect to the (log-concave) background distribution $\mathcal{D}$ which is uniform over the solid cube $[0,1]^{n}$.

We prove Theorem 36 using Theorem 35 and a reduction from Subset-Sum. Recall that an instance $W$ of the Subset-Sum problem is given by a non-negative integer $w_{0}$ and an $n$-tuple $w=(w_{1},\dots,w_{n})$ of non-negative integers, and the problem is to determine whether there is a subset $S\subseteq[n]$ such that $w_{0}=\sum_{i\in S}w_{i}.$ It is well known that there is a polynomial-time invertible reduction from Circuit-Sat to Subset-Sum in which a Circuit-Sat instance of size $n$ is mapped to a Subset-Sum instance in which each $w_{i}$ is at most $2^{n^{2}}.$ Since polynomial-time invertible Levin reductions compose, to prove Theorem 36 it suffices for us to give a polynomial-time invertible Levin reduction from Subset-Sum to the class of degree-2 PTFs which satisfies the properties stated in Theorem 35.

The reduction from Subset-Sum.

Given an instance $W=(w_{0},w)\in\mathbb{N}\times\mathbb{N}^{n}$ of Subset-Sum which has each $w_{i}\leq 2^{n^{2}}$, define the quadratic polynomial

[TABLE]

where $\lambda:=M\cdot{\lVert w{\rVert}}_{2}\gg 1$ for some $M=M(n)=cn$ to be determined later. The first observation is that the points achieving the minimum of $p_{W}$ over $[0,1]^{n}$ are in one-to-one correspondence with the solutions of $W$. Specifically, we observe the following.

Proposition 5.1**.**

Suppose that the Subset-Sum instance $W$ is satisfiable. Then the minimum of $p_{W}$ over $[0,1]^{n}$ is [math], and this minimum is achieved exactly at the points $x\in\{0,1\}^{n}$ satisfying $\left\langle w,x\right\rangle=w_{0}$.

Proof.

It is immediate to see that $p_{W}\geq 0$ on $[0,1]^{n}$, and further that for any solution of $W$ we have $p_{W}(x)=0$ (as the first term is zero by assumption, and the second is zero since $x\in\{0,1\}^{n}$). Conversely, consider any $x\in[0,1]^{n}$ such that $p_{W}(x)=0$. This implies both that $w\cdot x=w_{0}$ (since the first term has to be zero) and that $\sum_{i=1}^{n}x_{i}(1-x_{i})=0$, which since $x\in[0,1]^{n}$ yields $x\in\{0,1\}^{n}$. ∎

Based on Proposition 5.1, a natural choice of degree-two PTF for our hardness reduction would be, given an instance $W$, to take $f_{W}(x):=\mathrm{sign}(-p_{W}(x))$ (recall that $\mathrm{sign}(0)$ is defined to be 1). Indeed, in this case the satisfying assignments of $f_{W}$ in $[0,1]^{n}$ are in one-to-one correspondence with the solutions of the Subset-Sum instance, and given $x\in[0,1]^{n}\cap f_{W}^{-1}(1)$ it is trivial to produce the corresponding solution. We do not pursue exactly this simple approach, though, because of the issue discussed in Remark 7, namely that $[0,1]^{n}\cap f_{W}^{-1}(1)$ has measure zero under the uniform distribution on $[0,1]^{n}$ (as it is a finite set of discrete points). Since this set has measure zero, such an instance of the degree-2 PTF learning problem violates our requirement that the target function have a $2^{-n^{O(1)}}$ fraction of satisfying assignments under the background distribution.

We get around this issue by considering a slightly modified version of the above polynomial. More precisely, we will instead define

[TABLE]

and we will establish the following:

Proposition 5.2**.**

With $f_{W}$ defined as in Eq. 10, we have that

$[0,1]^{n}\cap f_{W}^{-1}(1)$* has measure at least $1/2^{O(n^{3})}$ under $\mathcal{U}_{[0,1]^{n}}$;* 2. 2.

$[0,1]^{n}\cap f_{W}^{-1}(1)$* is the disjoint union of $K$ sets $S_{1},\dots,S_{K}$, each with volume within a factor $2$ of each other under $\mathcal{U}_{[0,1]^{n}}$, where $K$ is the number of solutions to the Subset-Sum instance $W$ (and each set $S_{i}$ contains precisely one solution to $W$); and* 3. 3.

There is an efficient algorithm which, for any $j\in[K]$, given any $x\in S_{j}$, outputs a uniform sample drawn from $S_{j}$.

Part 1 of Proposition 5.2 addresses the “reasonableness” condition discussed in Remark 7. Part 2, together with our earlier discussion, is easily seen to give the desired polynomial-time invertible Levin reduction from Subset-Sum to degree-2 PTFs satisfying the properties stated in Theorem 35.

Towards proving Proposition 5.2 (which we will do at the end of this subsection), let us rewrite Eq. 9 as

[TABLE]

with

[TABLE]

Our next three claims will establish that (i) $f_{W}$ outputs $-1$ on any point too far from the Boolean hypercube (Section 5.1); (ii) $f_{W}$ outputs $-1$ on any point close to a point of the Boolean hypercube which is not a solution to the Subset-Sum instance $W$ (Section 5.1); and (iii) $f_{W}$ outputs $1$ on any point close to a point of the Boolean hypercube which is a solution to the Subset-Sum instance $W$ (Section 5.1). Combining the three (for suitable notions of “too close” and “too far” along with corresponding volume considerations will establish the proposition.

Define the parameter $\alpha_{n}$ as

[TABLE]

Claim \theclaim.

Suppose $x\in[0,1]^{n}$ is at $\ell_{1}$ distance more than $\alpha_{n}$ from every point from $\{0,1\}^{n}$. Then $f_{W}(x)=-1$.

Proof.

By symmetry, it is enough to consider the point $0_{n}\in\{0,1\}^{n}$. For any $x\in[0,1]^{n}$ such that $\alpha_{n}<{\lVert x{\rVert}}_{1}={\lVert x-0_{n}{\rVert}}_{1}<1/2$, we have

[TABLE]

by our choice of $\alpha_{n}$ as the solution to the equation $X(1-X)={\frac{1}{2\lambda}}$, and therefore $p_{W}(x)\geq\lambda B(x)>1/2$, so $f_{W}(x)=-1.$ ∎

Claim \theclaim.

Let $x\in[0,1]^{n}$, and suppose that $x$ is at $\ell_{1}$ distance at most $1/(4{\lVert w{\rVert}}_{2})$ of some $z\in\{0,1\}^{n}$ (thus necessarily unique). If $z$ is not a solution to the Subset-Sum instance $W$, then $f_{W}(x)=-1$.

Proof.

We can write

[TABLE]

the first inequality from the fact that all $w_{i}$’s are positive integers and $z\in\{0,1\}^{n}$, the third-to-last from Cauchy–Schwarz, and the second-to-last from monotonicity of $\ell_{p}$ norms. Therefore, we have $p_{W}(x)\geq s_{W}(x)\geq(3/4)^{2}>1/2$, and $f_{W}(x)=-1$. ∎

(Note that since $\alpha_{n}\operatorname*{\sim}\limits_{n\to\infty}\frac{1}{2M{\lVert w{\rVert}}_{2}}={o\!\left(1/{\lVert w{\rVert}}_{2}\right)}$, Section 5.1 obviously implies that any $x\in[0,1]$ which is at $\ell_{1}$ distance at most $\alpha_{n}$ from a non-solution point of $\{0,1\}^{n}$ must have $f_{W}(x)=-1$.)

Define the parameter $\beta_{n}$ as

[TABLE]

Claim \theclaim.

Let $x\in[0,1]^{n}$, and suppose that $x$ is at $\ell_{1}$ distance at most $\beta_{n}$ from some $z\in\{0,1\}^{n}$ (thus necessarily unique). If $z$ is a solution to the Subset-Sum instance $W$, then $f_{W}(x)=1$.

Proof.

We can write

[TABLE]

the last inequality again from Cauchy–Schwarz and monotonicity of $\ell_{p}$ norms. Moreover, we also have

[TABLE]

so that

[TABLE]

where the last inequality is a direct consequence of our choice of $\beta_{n}$ such that $\beta_{n}{\lVert w{\rVert}}_{2}$ is a solution to $X^{2}+MX=1/2$. This implies $f_{W}(x)=1$, as claimed. ∎

Proof of Proposition 5.2.

Sections 5.1 and 5.1 together imply (since, as noted after Section 5.1, $1/(4{\lVert w{\rVert}}_{2}\gg\alpha_{n}$) that if $x\in[0,1]^{n}$ is not within $\ell_{1}$ distance $\alpha_{n}$ of a solution $z\in\{0,1\}^{n}$ of the Subset-Sum instance $W$, then $f_{W}(x)=-1$. Moreover, Section 5.1 ensures that any $x\in[0,1]^{n}$ which is within $\ell_{1}$ distance $\beta_{n}$ of a solution $z\in\{0,1\}^{n}$ of the Subset-Sum instance $W$ satisfies $f_{W}(x)=1$.

Therefore, as $\alpha_{n}=o(1)$ and any two point in $\{0,1\}^{n}$ are at least at unit $\ell_{1}$ distance, the satisfying assignments $f_{W}^{-1}(1)$ are disjoint sets $S_{1},\dots,S_{K}$ (one around each of the $K$ satisfying assignments $z_{1},\dots,z_{K}\in\{0,1\}^{n}$ of the Subset-Sum instance) such that, for all $1\leq k\leq K$,

[TABLE]

This gives item (2) of the proposition, except for the assertion that any two $S_{i},S_{j}$ have volume within a factor of two of each other (we will establish this below).

For items (1) and (3), by symmetry it suffices to consider the case $z_{k}=0_{n}$. Noting that the volume of the $\ell_{1}$ ball of radius $r$ is $\operatorname*{Leb}(B^{1}_{n}(0_{n},r))=\frac{2^{n}r^{n}}{n!}$, and that restricting this ball to $[0,1]^{n}$ yields an $1/2^{n}$ fraction of the original volume, we get $\operatorname*{Leb}(B^{1}_{n}(0_{n},r)\cap[0,1]^{n})=\frac{r^{n}}{n!}$ and therefore by Sections 5.1 and 5.1 we have

[TABLE]

From Eqs. 12 and 13 and our setting of $\lambda=M{\lVert w{\rVert}}_{2}$, we get that $\alpha_{n},\beta_{n}$ are respectively $\frac{1}{2\lambda}+{O\left(\frac{1}{M\lambda}\right)}$ and $\frac{1}{2\lambda}-{O\left(\frac{1}{M\lambda}\right)}$, from which, as long as $M=cn$ (for some sufficiently big absolute constant $c>0$ to make $c^{\prime}>0$ below small enough), we get

[TABLE]

This shows that all volumes $\operatorname*{Leb}(S_{k})$ are within a constant factor $2$ of each other, finishing item 2 of the Proposition. Moreover, under the uniform distribution $\mathcal{U}_{[0,1]^{n}}$ over $[0,1]^{n}$, the above along with our bound ${\lVert w{\rVert}}_{\infty}\leq 2^{n^{2}}$ and the choice of $\lambda=M\cdot{\lVert w{\rVert}}_{2}$ implies that

[TABLE]

as $\lambda\leq M\sqrt{n}{\lVert w{\rVert}}_{\infty}=O(n^{3/2}2^{n^{2}})$.

This establishes item 1 of Proposition 5.2, so it remains only to show that item 3 holds. To establish this last item (i.e., that for any $k$ one can efficiently, given any $x^{\ast}\in S_{k}$, sample uniformly from $S_{k}$) it suffices to observe that (i) given such a $x^{\ast}$, it is immediate to find the corresponding satisfying assignment $z_{k}\in\{0,1\}^{n}$ (by rounding the coordinates); (ii) efficient sampling uniformly from $B^{1}_{n}(z_{k},\alpha_{n})$ can be done using elementary techniques; and (iii) since the volume of $S_{k}$ is within a factor of $2$ of the volume of $B^{1}_{n}(z_{k},\alpha_{n})$ and $f_{W}$ can be efficiently evaluated, rejection sampling allows us to sample uniformly from $S_{k}$ in an efficient way. This concludes the proof of Proposition 5.2 and hence also of Theorem 36. ∎

5.2 A lower bound for degree-4 PTFs under the normal distribution

In this subsection, we prove Theorem 4, which shows that even under a very strong assumption on the ambient distribution one cannot hope for much better than Theorem 3:

Theorem 37** (Theorem 4, restated).**

Under 1, there is no subexponential-time algorithm $A$ for learning the class $\mathsf{PTF}^{n}_{4}$ of $n$-variable degree-four polynomial threshold functions with respect to the standard normal distribution $\mathcal{N}(0,1)^{n}$.

The rest of this section is dedicated to the proof of this theorem. As for Theorem 2, our starting point is a reduction from Subset-Sum, although a slightly different one. First off, it will be convenient for us to now view the Subset-Sum problem as being over $\{-1,1\}^{n}$ rather than $\{0,1\}^{n}$: i.e., the goal is to determine whether there is a string $z\in\{-1,1\}^{n}$ such that $w\cdot z=w_{0}$. (It is easy to see that this is equivalent to the original problem over $\{0,1\}^{n}$.) Now, given an instance $W=(w_{0},w)\in\mathbb{N}\times\mathbb{N}^{n}$ of this form of Subset-Sum in which each $w_{i}\leq 2^{n^{2}}$, we define the quartic polynomial

[TABLE]

where $\lambda:=cn\max({\lVert w{\rVert}}_{2}^{2},n)\gg 1$ for some sufficiently large absolute constant $c>0$, and we define

[TABLE]

as in Theorem 2. Compared to the degree-2 lower bound, the “penalization” term $B(x)$ is now a degree-4 polynomial, whose role is to enforce that satisfying assignments to $f_{W}$ from $\mathbb{R}^{n}$ must be “essentially Boolean”, where now “Boolean” means “belonging to $\{-1,1\}^{n}$” (note that $B$ vanishes exactly on $\{-1,1\}^{n}$) over inputs from $\mathbb{R}^{n}$).

Analogously to the proof of Theorem 2, we establish the facts that (1) zeros of $p_{W}$ are exactly the solutions to the Subset-Sum instance $W$; (2) if a point is far from a Boolean point, then it does not satisfy $f_{W}$; (3) if a point is close to a Boolean point not satisfying $W$, then it does not satisfy $f_{W}$; and (4) if a point is close to a Boolean point satisfying $W$, then it does satisfy $f_{W}$. Altogether, this will in turn allow us to establish the proposition below, which is closely analogous to Proposition 5.2:

Proposition 5.3**.**

With $f_{W}$ defined as in Eq. 17, we have that

$f_{W}^{-1}(1)$* has measure at least $1/2^{O(n^{3})}$ under $\mathcal{N}(0,1)^{n}$;* 2. 2.

$f_{W}^{-1}(1)$* is the disjoint union of $K$ sets $S_{1},\dots,S_{K}$, each with measure within a factor $2$ of each other under $\mathcal{N}(0,1)^{n}$, where $K$ is the number of solutions to the Subset-Sum instance $W$ (and each set $S_{i}$ contains precisely one solution to $W$); and* 3. 3.

There is an efficient algorithm which, for any $j\in[K]$, given any $x\in S_{j}$, outputs a draw from ${\cal N}(0,1)^{n}$ restricted to $S_{j}$.

Similar to Proposition 5.2, Proposition 5.3 is easily seen to satisfy the “reasonableness” condition and give the desired polynomial-time invertible Levin reduction from Subset-Sum to degree-2 PTFs satisfying the properties stated in Theorem 35. The following is entirely analogous to Proposition 5.1:

Proposition 5.4**.**

Suppose that the Subset-Sum instance $W$ is satisfiable. Then the minimum of $p_{W}$ over $\mathbb{R}^{n}$ is [math], and this minimum is achieved exactly at the points $x\in\{-1,1\}^{n}$ satisfying $\left\langle w,x\right\rangle=w_{0}$.

Proof.

For any solution $x$ of $W$ we have $p_{W}(x)=0$ (as the first term $s_{W}(x)$ is zero by assumption, and the penalization term $B(x)$ is zero since $x\in\{-1,1\}^{n}$). Conversely, consider any $x\in\mathbb{R}^{n}$ such that $p_{W}(x)=0$. Since $s_{W},B\geq 0$, this implies both that $w\cdot x=w_{0}$ (since the first term has to be zero) and that $\sum_{i=1}^{n}(x_{i}^{2}-1)^{2}=0$, which yields $x\in\{-1,1\}^{n}$. ∎

As before, our actual construction works with the PTF $\mathrm{sign}({\frac{1}{2}}-p_{W}(x))$ rather than $\mathrm{sign}(-p_{W}(x))$ because of the requirement that the learning problems under consideration have at least an inverse exponential fraction of probability mass under the background distribution lying on satisfying assignments of the target function (recall Remark 7). Define the parameter $\alpha_{n}$ as the smallest positive solution to to $4(1-X)^{2}X^{2}\geq 1/(2\lambda)$, i.e.,

[TABLE]

Claim \theclaim.

Suppose $x\in\mathbb{R}^{n}$ is at $\ell_{2}$ distance more than $\alpha_{n}$ from every point from $\{-1,1\}^{n}$. Then $f_{W}(x)=-1$.

Proof.

For any such $x\in\mathbb{R}^{n}$, we have that the closest Boolean point to $x$ is $x^{\ast}\in\{-1,1\}^{n}$, defined by $x^{\ast}_{i}=\mathrm{sign}(x_{i})$ for every $i\in[n]$. Then,

[TABLE]

Since $(x_{i}+x^{\ast}_{i})^{2}\geq 1$ for all $i$, we first observe that if there exists some $i\in[n]$ such that $\left\lvert x_{i}-x^{\ast}_{i}\right\rvert>2\alpha_{n}$, then $B(x)>4\alpha_{n}^{2}>1/(2\lambda)$, and we are done. Thus we may assume that $\left\lvert x_{i}-x^{\ast}_{i}\right\rvert<2\alpha_{n}$ for all $i$. It follows that

[TABLE]

where the first inequality uses $\left\lvert x_{i}-x^{\ast}_{i}\right\rvert<2\alpha_{n}$ and the last inequality follows by recalling that ${\lVert x-x^{\ast}{\rVert}}_{2}^{2}>\alpha_{n}^{2}$. We then get $p_{W}(x)\geq\lambda B(x)>1/2$, so $f_{W}(x)=-1.$ ∎

Claim \theclaim.

Let $x\in\mathbb{R}^{n}$, and suppose that $x$ is at $\ell_{2}$ distance at most $1/(4{\lVert w{\rVert}}_{2})$ from some $z\in\{-1,1\}^{n}$ (thus necessarily unique). If $z$ is not a solution to the Subset-Sum instance $W$, then $f_{W}(x)=-1$.

Proof.

The proof is almost identical to that of Section 5.1. We write

[TABLE]

the first inequality from the fact that all $w_{i}$’s are positive integers and $z\in\{-1,1\}^{n}$, the third from Cauchy–Schwarz. Therefore, we have $p_{W}(x)\geq s_{W}(x)\geq(3/4)^{2}>1/2$, and $f_{W}(x)=-1$. ∎

Define the parameter $\beta_{n}$ as the smallest positive solution to $({\lVert w{\rVert}}_{2}^{2}+\lambda(2+X)^{2})X^{2}=1/2$, so that

[TABLE]

Claim \theclaim.

Let $x\in\mathbb{R}^{n}$, and suppose that $x$ is at $\ell_{2}$ distance at most $\beta_{n}$ from some $z\in\{-1,1\}^{n}$ (thus necessarily unique). If $z$ is a solution to the Subset-Sum instance $W$, then $f_{W}(x)=1$.

Proof.

We can write

[TABLE]

the second-to-last inequality again from Cauchy–Schwarz. Moreover, we also have

[TABLE]

where we used the fact that $\left\lvert x_{i}+z_{i}\right\rvert\leq 2+\beta_{n}$ for all $i$ (since otherwise ${\lVert x-z{\rVert}}_{2}>\beta_{n}$). Therefore,

[TABLE]

where the last inequality is due to our choice of $\beta_{n}$ in (19). This implies $f_{W}(x)=1$, as claimed. ∎

Proof of Proposition 5.3.

Sections 5.2 and 5.2 together imply (since $1/(4{\lVert w{\rVert}}_{2}\gg\alpha_{n}$) that if $x\in\mathbb{R}^{n}$ is not within $\ell_{2}$ distance $\alpha_{n}$ of a solution $z\in\{-1,1\}^{n}$ of the Subset-Sum instance $W$, then $f_{W}(x)=-1$. Moreover, Section 5.2 ensures that any $x\in\mathbb{R}^{n}$ which is within $\ell_{2}$ distance $\beta_{n}$ of a solution $z\in\{-1,1\}^{n}$ of the Subset-Sum instance $W$ satisfies $f_{W}(x)=1$.

Therefore, as $\alpha_{n}=o(1)$ and any two points in $\{-1,1\}^{n}$ are at least at unit $\ell_{2}$ distance, the satisfying assignments $f_{W}^{-1}(1)$ are disjoint sets $S_{1},\dots,S_{K}$ (one around each of the $K$ satisfying assignments $z_{1},\dots,z_{K}\in\{-1,1\}^{n}$ of the Subset-Sum instance) such that, for all $1\leq k\leq K$,

[TABLE]

This gives the first part of item (2) of the proposition.

For items (1) and (3) (and the last part of item (2)), by symmetry it suffices to consider the case $z_{k}=1_{n}$: we want to estimate the Gaussian measure of the $\ell_{2}$ ball of radius $r$, $\phi(B^{2}_{n}(1_{n},r))$. For $r=O(1/\sqrt{n})$, we claim that this is within constant factors of $\phi(1_{n})\cdot\operatorname*{Leb}(B^{2}_{n}(1_{n},r))$, i.e., we claim that

[TABLE]

This is because, for such a small radius $r$, the Gaussian density changes by at most a constant factor within the radius-$r$ $\ell_{2}$ ball centered at $1_{n}$: indeed, for any $z$ in this ball, ${\lVert z{\rVert}}_{2}=\sqrt{n}\pm r$, so that $e^{-{\lVert z{\rVert}}_{2}^{2}/2}=e^{-n/2\pm O(1)}={\Theta\left(e^{-n/2}\right)}$. Therefore, by the above we have

[TABLE]

From Eqs. 18 and 19 and our setting of $\lambda=cn{\lVert w{\rVert}}_{2}^{2}$, we get that $\alpha_{n},\beta_{n}$ are respectively $\frac{1}{\sqrt{8\lambda}}+{O\left(\frac{1}{\lambda}\right)}$ and $\frac{1}{\sqrt{8\lambda}}-{O\left(\frac{1}{\lambda}\right)}$, from which, because $\lambda\gg n^{2}$, we get

[TABLE]

This shows that all measures $\phi(S_{k})$ are within a constant factor of each other. Moreover, under the Gaussian measure $\phi$ over $\mathbb{R}^{n}$, the above along with our bound ${\lVert w{\rVert}}_{\infty}\leq 2^{n^{2}}$ and the choice of $\lambda=cn\max({\lVert w{\rVert}}_{2}^{2},n)$ implies that

[TABLE]

as $\lambda=O(n^{2}2^{2n^{2}})$.

This establishes item 1 of Proposition 5.3, so it remains only to show that item 3 holds. To establish this last item (i.e., that for any $k$ one can efficiently, given any $x^{\ast}\in S_{k}$, sample according to the Gaussian measure restricted to $S_{k}$) it suffices to observe that (i) given such a $x^{\ast}$, it is immediate to find the corresponding satisfying assignment $z_{k}\in\{-1,1\}^{n}$ (by rounding the coordinates); (ii) efficient sampling uniformly from $B^{2}_{n}(z_{k},\alpha_{n})$ can be done using elementary techniques; and (iii) since the volume of $S_{k}$ is within a constant factor of the volume of $B^{2}_{n}(z_{k},\alpha_{n})$ and $f_{W}$ can be efficiently evaluated, and the Gaussian pdf can be easily computed on any point in $S_{k}$ and this value varies only by an $O(1)$ factor between any two points in $S_{k}$, rejection sampling allows us to sample according to the Gaussian measure restricted to $S_{k}$ in an efficient way. This concludes the proof of Proposition 5.3 and hence also of Theorem 37. ∎

Appendix A Hardness of distribution learning with respect to arbitrary background distributions

In this appendix we justify our focus on structured continuous distributions (namely Gaussian and log-concave) by proving that for general distributions ${\cal D}$, learning $\mathcal{D}_{f}$ even for very simple functions $f$ (indeed, linear threshold functions) is computationally hard.

We actually prove an even stronger hardness result, by assuming both that

• •

the background distribution $\mathcal{D}$ is provided to the learning algorithm via an evaluation oracle to a function $\tilde{\mathcal{D}}$ such that on any input $x$, $\tilde{\mathcal{D}}(x)$ is proportional to $\mathcal{D}(x)$; and that

• •

the algorithm has access to an exact sampler for $\mathcal{D}$.

(This is at least as strong as the model we use for our positive results on log-concave distributions, presented in Section 2, in which the learner only receives an evaluation oracle for some measure which is a rescaling of the probability density function.) We show that, even in this setting, any distribution learning algorithm with respect to an arbitrary background distribution $\mathcal{D}$ may need to take exponential time, even if the function $f$ is guaranteed to be a halfspace which is provided to the learning algorithm.

To see this, suppose that the distribution $\mathcal{D}$ is $\mathcal{U}_{g}$ (the uniform distribution over $\{-1,1\}^{n}$ restricted to the satisfying assignments of $g$), where $g$ is a halfspace whose description is provided to the algorithm. We note that the distribution $\mathcal{D}$ satisfies both our hypothesis. Namely,

• •

The 0/1-valued function $g(x)$ is proportional to $\mathcal{D}(x)$, and $g(x)$ can be efficiently computed. (In fact, the result of Dyer [Dye03] (and earlier work of Morris and Sinclair [MS04]) also establishes that $\mathcal{D}(x)$ itself can be efficiently evaluated to a multiplicative factor of $(1+\varepsilon)$ in $\mathrm{poly}(n/\varepsilon)$ time.)

• •

By the result of Dyer [Dye03]), there is an efficient algorithm to exactly sample from $\mathcal{D}$.

Now let us define $h(x)=f(x)\wedge g(x)$, so $h$ is an intersection of two halfspaces. Note that the distribution $\mathcal{U}_{h}$ is exactly $\mathcal{D}_{f}$. However, under Assumption 1, it was shown in [DDS15a] that there is no algorithm with running time $2^{n^{\delta}}$ (for some fixed $\delta>0$) to learn $\mathcal{U}_{h}$ given samples from the distribution (Theorem 1.5 of [DDS15a], see also Theorem 62 of [DDS15b]). It can be verified that this hardness result continues to hold even if the description of both halfspaces $f$ and $g$ is given to the algorithm. This implies that under Assumption 1, there is no algorithm with running time $2^{n^{\delta}}$ to learn $\mathcal{D}_{f}$ given samples from the distribution. Now, since $g$ is known to the algorithm, by the above discussion the algorithm can efficiently (i) sample from $\mathcal{D}$, and (ii) evaluate the function $g(x)$ which is proportional to $\mathcal{D}(x)$. This shows that for arbitrary distributions $\mathcal{D}$ over the discrete cube, learning $\mathcal{D}_{f}$ is computationally hard for a halfspace $f$ even when we have access to a sampler for $\mathcal{D}$ as well as an evaluation oracle to $\mathcal{D}$.

Finally, observe that this example straightforwardly generalizes to continuous probability distributions by replacing the points of $\{-1,1\}^{n}$ in the support by disjoint (suitably small) balls centered at those points, and making $\mathcal{D}$ uniform on the union of these disjoint balls (akin to the constructions of Section 5).

Appendix B Smooth version of $\tilde{\mu}$

For technical reasons related to the details of the literature on sampling from log-concave distributions, it is helpful for us to be able to work with a “cleaned-up” version of the measure $\tilde{\mu}$, which we denote $\tilde{\mu}_{1}$. The associated distribution $\mu_{1}$ is entirely supported on ${\cal B}(0,n^{C})$, puts at least some minimum amount of mass on every point in ${\cal B}(0,n^{C})$, and is statistically very close to the actual distribution $\mu$. The precise statement we will use is stated and proved below.

Theorem 38**.**

Let $\tilde{\mu}$ be a measure such that the associated density $\mu$ satisfies conditions C1, C2 and C3 from Section 2. Let $\tilde{\mu}_{1}\colon\mathbb{R}^{n}\to(0,\infty)$ be the measure which is defined as follows:

[TABLE]

Let $\mu_{1}$ be the density corresponding to $\tilde{\mu}_{1}$. Then,

$\mu_{1}$* is log-concave.* 2. 2.

For any $x\in B^{2}(0,n^{C})$, $\mu_{1}(x)\geq 2^{-2n^{C}-5n-3}\cdot n^{-{2}Cn}$ 3. 3.

$\|\mu-\mu_{1}\|_{1}\leq 2^{-\frac{3n}{2}}$.

Proof.

The first item follows simply from the fact that restricting $\tilde{\mu}$ to a convex set preserves log-concavity as does adding a constant function.

To prove the next item, observe that

[TABLE]

We now make two observations:

1. (i)

By C2, we have that $\tilde{\mu}_{T}(x)$ is at most $2^{n^{C}}$ for all $x$. Since the volume of $B^{2}(0,n^{C})$ is at most $2^{n}\cdot n^{Cn}$, the first integral in Equation 22 is at most $2^{n+n^{C}}\cdot n^{Cn}$. 2. 1.

Since $n\geq 1$ and $C\geq 2$, $2^{1-Cn}\leq 1/2$.

Combining these two observations with (22), we have $\int_{x}\tilde{\mu}(x)\leq 2^{n^{C}+n+2}\cdot n^{Cn},$ and hence $\int_{x}\tilde{\mu}_{1}(x)\leq 2^{n^{C}+n+3}$ (with room to spare). This immediately implies that for any $x\in B^{2}(0,n^{C})$,

[TABLE]

This proves the second item. To prove the third item, first observe that if $\mu_{T}$ denotes the density induces by the measure $\tilde{\mu}_{T}$, then

[TABLE]

Here the penultimate inequality uses that $\tilde{\mu}_{T}(x)\leq\tilde{\mu}(x)$ as well as condition C3, and the last inequality uses condition C2. Consequently, if $\mathcal{A}=\{x:\mu_{T}(x)\not=0\}$, then since $\mu_{T}$ is a distribution, it must hold that $\mathsf{Vol}(\mathcal{A})\geq 2^{-n-1}$. We thus have that

[TABLE]

where the last inequality is by the definition of $\tilde{\mu}_{T}$ as given in condition C3 (truncating $n^{C}$ bits after the binary point). Again using that $\tilde{\mu}(x)\geq\tilde{\mu}_{T}(x)$, we obtain

[TABLE]

Further, we also have that

[TABLE]

where the last inequality is by (23). With this, we are ready to prove the third item.

[TABLE]

Here the last inequality uses the definitions of $\mu$ and $\mu_{1}$ and the triangle inequality. We now bound the terms in Equation 25 individually. We start with the first term.

[TABLE]

To bound the second term, we have

[TABLE]

Combining this with (26) and plugging back into (25), we get

[TABLE]

The above assumes that $n\geq 5$. This finishes the proof.

∎