Differentially Private Learning of Geometric Concepts

TL;DR

This paper introduces differentially private algorithms for efficiently learning unions of polygons in the plane, achieving PAC learning and privacy guarantees with a sample size depending on the number of edges and domain size.

Contribution

It provides the first differentially private algorithms for learning unions of polygons with theoretical guarantees on sample complexity.

Findings

Achieves $(eta,eta)$-PAC learning with privacy guarantees.

Sample complexity scales with the number of edges and domain size.

Algorithms are efficient and applicable to non-convex polygon unions.

Abstract

We present differentially private efficient algorithms for learning union of polygons in the plane (which are not necessarily convex). Our algorithms achieve $(\alpha,\beta)$-PAC learning and $(\epsilon,\delta)$-differential privacy using a sample of size $\tilde{O}\left(\frac{1}{\alpha\epsilon}k\log d\right)$, where the domain is $[d]\times[d]$ and $k$ is the number of edges in the union of polygons.

Equations

$$\mathrm{Pr}[\mathcal{A}(S)\in T]\le e^{\epsilon }\cdot \mathrm{Pr}[\mathcal{A}(S^{\prime })\in T]+\delta .$$

$$\mathrm{Pr}\left[{\mathrm{error}}_{\mathcal{D}}(c,h)>\alpha \right]\le \beta .$$

$$\Pr\Big{[}\exists h\in C\text{ s.t.\ }{\rm error}_{\mathcal{D}}(h,c)>\alpha\ \land\ {\rm error}_{S}(h)\leq\frac{\alpha}{2}\Big{]}\leq\beta.$$

$$\mathrm{VC}(H^{\vee k})\le O(k\log (k)\cdot \mathrm{VC}(H))\text{and}\mathrm{VC}(H^{\wedge k})\le O(k\log (k)\cdot \mathrm{VC}(H)).$$

$$\mathrm{Pr}[\mathcal{A}(S)\in T]\le e^{\epsilon }\cdot \mathrm{Pr}[\mathcal{A}(S^{\prime })\in T]+\delta ,$$

$$\Pr\Big{[}|\mathcal{A}(S)-f(S)|>\Delta\Big{]}\leq\exp\left(-\frac{\epsilon\Delta}{s}\right).$$

$${\#}_{h\to 0}(S^1)=|\{x_i\in S^1:h(x_i)=0\}|\text{and}{\#}_{h\to 0}(S^0)=|\{x_i\in S^0:h(x_i)=0\}|.$$

$${\#}_{h_j\to 0}(S^0)\ge \frac{b_j}{k}-\lambda \ge \frac{|S^0|-2\Delta }{k}-\lambda \ge \frac{|S^0|}{2k}.$$

$$n=\Theta \left(\frac{k\log \frac{1}{\alpha }}{\alpha }\left(\mathrm{VC}(H)\log (k)+\lambda +\frac{1}{\epsilon }\log \left(\frac{k}{\beta }\log \frac{1}{\alpha }\right)\right)\right).$$

$$q(j)\gtrsim \underset{f\in H}{\max }\{q(f)\}-\frac{1}{\hat{\epsilon }}\log |H|\approx \underset{f\in H}{\max }\{q(f)\}-\frac{\sqrt{k}}{\epsilon }\log |H|.$$

$$\frac{\mathrm{Pr}[\text{SetCoverLearner}(S)=\vec{h}]}{\mathrm{Pr}[\text{SetCoverLearner}(S^{\prime })=\vec{h}]}.$$

$$\frac{\mathrm{Pr}[\text{SetCoverLearner}(S)=\vec{h}|\vec{w}]}{\mathrm{Pr}[\text{SetCoverLearner}(S^{\prime })=\vec{h}|\vec{w}]}$$

$$\frac{\mathrm{Pr}[\text{SetCoverLearner}(S)=\vec{h}]}{\mathrm{Pr}[\text{SetCoverLearner}(S^{\prime })=\vec{h}]}\le \exp (\hat{\epsilon }).$$

$$\frac{\mathrm{Pr}[\text{SetCoverLearner}(S)=\vec{h}|\vec{w}]}{\mathrm{Pr}[\text{SetCoverLearner}(S^{\prime })=\vec{h}|{\vec{w}}^{\prime }]}$$

$$\le \prod _{j=1}^t\left(\frac{(\exp (\hat{\epsilon })-1)\cdot \sum _{\begin{array}{c}f\in H:\\ f(x^{\ast })=0\end{array}}\exp (\hat{\epsilon }\cdot q_{j,S,w_j}(f))+\sum _{f\in H}\exp (\hat{\epsilon }\cdot q_{j,S,w_j}(f))}{\sum _{f\in H}\exp (\hat{\epsilon }\cdot q_{j,S,w_j}(f))}\right)$$

$$=\prod _{j=1}^t\left(1+(\exp (\hat{\epsilon })-1)\cdot p_j(S,w_j)\right)$$

$$\sum _{j=1}^{t-1}{p}_j(S,w_j)\le \ln (1/\delta ).$$

$$\frac{\mathrm{Pr}[\text{SetCoverLearner}(S)=\vec{h}|\vec{w}]}{\mathrm{Pr}[\text{SetCoverLearner}(S^{\prime })=\vec{h}|{\vec{w}}^{\prime }]}$$

$$\mathrm{Pr}[\text{SetCoverLearner}(S)=\vec{h}]$$

$$\displaystyle=\sum_{\begin{subarray}{c}w_{1},\dots,w_{t},\\ w_{t+1},\dots,w_{2k\log\frac{2}{\alpha}}\end{subarray}}\Pr\left[\begin{array}[]{l}w_{1},\dots,w_{t},\\ w_{t+1},\dots,w_{2k\log\frac{2}{\alpha}}\end{array}\right]\cdot\Pr\left[\texttt{SetCoverLearner}(S)=\vec{h}\left|\begin{array}[]{l}w_{1},\dots,w_{t},\\ w_{t+1},\dots,w_{2k\log\frac{2}{\alpha}}\end{array}\right.\right]$$

$$\displaystyle=\sum_{\begin{subarray}{c}w_{1},\dots,w_{t},\\ w_{t+1},\dots,w_{2k\log\frac{2}{\alpha}}\end{subarray}}\Pr\left[\begin{array}[]{l}w_{1}+1,\dots,w_{t}+1,\\ w_{t+1},\dots,w_{2k\log\frac{2}{\alpha}}\end{array}\right]\cdot\Pr\left[\texttt{SetCoverLearner}(S)=\vec{h}\left|\begin{array}[]{l}w_{1}+1,\dots,w_{t}+1,\\ w_{t+1},\dots,w_{2k\log\frac{2}{\alpha}}\end{array}\right.\right]$$

$$\displaystyle\leq\sum_{\begin{subarray}{c}w_{1},\dots,w_{t},\\ w_{t+1},\dots,w_{2k\log\frac{2}{\alpha}}\end{subarray}}e^{\varepsilon}\cdot\Pr\left[\begin{array}[]{l}w_{1},\dots,w_{t},\\ w_{t+1},\dots,w_{2k\log\frac{2}{\alpha}}\end{array}\right]\cdot\Pr\left[\texttt{SetCoverLearner}(S)=\vec{h}\left|\begin{array}[]{l}w_{1}+1,\dots,w_{t}+1,\\ w_{t+1},\dots,w_{2k\log\frac{2}{\alpha}}\end{array}\right.\right]$$

$$\displaystyle\leq\sum_{\begin{subarray}{c}w_{1},\dots,w_{t},\\ w_{t+1},\dots,w_{2k\log\frac{2}{\alpha}}\end{subarray}}e^{\varepsilon}\cdot\Pr\left[\begin{array}[]{l}w_{1},\dots,w_{t},\\ w_{t+1},\dots,w_{2k\log\frac{2}{\alpha}}\end{array}\right]\cdot e^{\varepsilon}\cdot\Pr\left[\texttt{SetCoverLearner}(S^{\prime})=\vec{h}\left|\begin{array}[]{l}w_{1},\dots,w_{t},\\ w_{t+1},\dots,w_{2k\log\frac{2}{\alpha}}\end{array}\right.\right]$$

$$=e^{2\epsilon }\cdot \mathrm{Pr}[\text{SetCoverLearner}(S^{\prime })=\vec{h}]$$

$$\mathrm{Pr}[\text{SetCoverLearner}(S)\in F]=\sum _{\vec{f}\in F}\mathrm{Pr}\left[\text{SetCoverLearner}(S)=\vec{f}\right]$$

$$=\sum _{\begin{array}{c}\vec{f}\in F:\vec{f}\text{ is}\\ \ln (1/\delta )\text{-bad}\\ \text{for }S\end{array}}\mathrm{Pr}\left[\text{SetCoverLearner}(S)=\vec{f}\right]+\sum _{\begin{array}{c}\vec{f}\in F:\vec{f}\text{ is}\\ \ln (1/\delta )\text{-good}\\ \text{for }S\end{array}}\mathrm{Pr}\left[\text{SetCoverLearner}(S)=\vec{f}\right]$$

$$\le \delta +\sum _{\begin{array}{c}\vec{f}\in F:\vec{f}\text{ is}\\ \ln (1/\delta )\text{-good }\text{for }S\end{array}}{e}^{2\epsilon }\cdot \mathrm{Pr}\left[\text{SetCoverLearner}(S^{\prime })=\vec{f}\right]$$

$$\le e^{2\epsilon }\cdot \mathrm{Pr}[\text{SetCoverLearner}(S^{\prime })\in F]+\delta .$$

$$\mathrm{HALFPLANE}=\{f_{a,b,c}:a,b,c\in \mathbb{R}\},\text{where }{f}_{a,b,c}(x,y)=1\text{ iff }cy\ge ax+b.$$

$$\mathtt{CONVEX}-\mathtt{k}-\mathtt{GON}=\stackrel{\wedge k}{\mathrm{HALFPLANE}}.$$

$$\underset{d}{\mathrm{HALFPLANE}}=\left\{f_{\hat{a},b}:-2d^2\le \hat{a}\le 6d^2,-2d^2\le b\le 2d^2\right\},$$

$$\text{where }{f}_{\hat{a},b}(x,y)=1\text{ iff }zy\ge z(ax+b)\text{ for }a=\hat{a}-4d^2\cdot \text{mathbbm}{1}_{\{\hat{a}>2d^2\}}\text{ and }z=1-2\cdot \text{mathbbm}{1}_{\{\hat{a}>2d^2\}}.$$

$$y=\frac{y_2-y_1}{x_2-x_1}x+\left(y_1-x_1\frac{y_2-y_1}{x_2-x_1}\right)\triangleq ax+b,$$

$$y\ge a_{1}x+b_1\text{iff}y\ge a_{2}x+b_2,$$

$${\int }_{F\times D}p(a,b){\mathrm{d}}^2(a,b)$$

Full text

1 Introduction

Machine learning algorithms have exciting and wide-range potential. However, as the data frequently contain sensitive personal information, there are real privacy concerns associated with the development and the deployment of this technology. Motivated by this observation, the line of work on differentially private learning (initiated by [23]) aims to construct learning algorithms that provide strong (mathematically proven) privacy protections for the training data. Both government agencies and industrial companies have realized the importance of introducing strong privacy protection to statistical and machine learning tasks. A few recent examples include Google [20] and Apple [27] that are already using differentially private estimation algorithms that feed into machine learning algorithms, and the US Census Bureau announcement that they will use differentially private data publication techniques in the next decennial census [1]. Differential privacy is increasingly accepted as a standard for rigorous privacy. We refer the reader to the excellent surveys in [17] and [28]. The definition of differential privacy is,

Definition 1.1** ([16]).**

Let $\mathcal{A}$ be a randomized algorithm whose input is a sample. Algorithm $\mathcal{A}$ is $(\varepsilon,\delta)$-differentially private if for every two samples $S,S^{\prime}$ that differ in one example, and for any event $T$, we have

[TABLE]

For now, we can think of a (non-private) learner as an algorithm that operates on a set of classified random examples, and outputs a hypothesis $h$ that misclassifies fresh examples with probability at most (say) $\tfrac{1}{10}$. A private learner must achieve the same goal while guaranteeing that the choice of $h$ preserves differential privacy of the sample points. Intuitively this means that the choice of $h$ should not be significantly affected by any particular sample.

While many learning tasks of interest are compatible with differential privacy, privacy comes with a cost (in terms of computational resources and the amount of data needed), and it is important to understand how efficient can private learning be. Indeed, there has been a significant amount of work aimed at understanding the sample complexity of private learning [4, 12, 5, 6, 21, 10, 2], the computational complexity of private learning [11], and studying variations of the private learning model [7, 9, 13, 3]. However, in spite of the significant progress made in recent years, much remains unknown and answers to fundamental questions are still missing. In particular, the literature lacks effective constructions of private learners for specific concept classes of interest, such as halfspaces, polynomials, $d$-dimensional balls, and more. We remark that, in principle, every (non-private) learner that works in the statistical queries (SQ) model of Kearns [24] can be transformed to preserve differential privacy. However, as the transformation is only tight up to polynomial factors, and as SQ learners are often much less efficient than their PAC learners counterparts, the resulting private learners are typically far from practical.

In this work we make an important step towards bringing differentially private learning closer to practice, and construct an effective algorithm for privately learning simple geometric shapes, focusing on polygons in the plane. To motivate our work, consider the task of analyzing GPS navigation data, or the task of learning the shape of a flood or a fire based on users’ location reports. As user location data might be sensitive, the ability to privately learn such shapes is of significant importance.

1.1 A Non-Private Learner for Conjunctions and Existing Techniques

Our learner is obtained by designing a private variant for the classical (non-private) learner for conjunctions using the greedy algorithm for set-cover. Before describing our new learner, we first quickly recall this non-private technique (see e.g., [25] for more details).

Let $\mathop{\rm{CONJ}}\nolimits_{k,d}$ denote the class of all conjunctions (i.e., AND) of at most $k$ literals over $d$ Boolean variables $v_{1},\dots,v_{d}$, e.g., $v_{1}\wedge\overline{v}_{4}\wedge v_{5}$. Here, for a labeled example $(\vec{x},\sigma)$, the vector $\vec{x}\in\{0,1\}^{d}$ is interpreted as an assignment to the $d$ Boolean variables, and $\sigma=1$ iff this assignment satisfies the target concept. Given a sample $S=\{(\vec{x}_{i},\sigma_{i})\}$ of labeled examples, the classical (non-private) learner for this class begins with the hypothesis $h=v_{1}\wedge\overline{v}_{1}\wedge\dots\wedge v_{d}\wedge\overline{v}_{d}$, and then proceeds by deleting from $h$ any literal that “contradicts” a positively labeled example in $S$. Observe that at the end of this process, the set of literals appearing in $h$ contains the set of literals appearing in the target concept (because a literal is only deleted when it is contradicted by a positively labeled example).

The next step is to eliminate unnecessary literals from the hypothesis $h$ (in order to guarantee generalization). Note that removing literals from $h$ might cause it to err on negative example in $S$, and hence, the algorithm must carefully choose which of the literals to eliminate. This can be done using the greedy algorithm for set cover as follows. We have already made sure that each of the literals in $h$ does not err on positive examples in $S$ (since such literals were deleted), and we know that there is a choice of $k$ literals from $h$ that together correctly classify all negative examples in $S$ (since we know that the $k$ literals of the target concept are contained in $h$). Thus, our task can be restated as identifying a small number of literals from $h$ that together correctly classify all negative examples in $S$. This can be done using the greedy algorithm for set cover, where every literal in $h$ corresponds to a set, and this set “covers” a negative example if the literal is zero on this example.

To summarize, the algorithm first identifies the collection of all literals that are consistent with the positive data, and then uses the greedy algorithm for set cover in order to identify a small subset of these literals that together correctly classify the negative data. This is a good starting point for designing a private learner for conjunctions, since the greedy algorithm for set cover has a private variant [22]. The challenge here is that in the private algorithm of Gupta et al. [22], the collection of sets from which the cover is chosen is assumed to be fixed and independent of the input data. In contrast, in our case the collection of sets corresponds to the literals that correctly classified the positive data, which is data dependent. One might try to overcome this challenge by first identifying, in a private manner, a collection $L$ of all literals that correctly classify (most of) the positive data, and then to run the private algorithm of Gupta et al. [22] to choose a small number of literals from $L$. However, a direct implementation of this idea would require accounting for the privacy loss incurred due to each literal in $L$. As $|L|$ can be linear in $d$ (the number of possible literals), this would result in an algorithm with sample complexity $\mathop{\rm{poly}}\nolimits(d)$. When we apply this strategy to learn polygons in the plane, $d$ will correspond to the size of an assumed grid on the plane, which we think of as being very big, e.g., $d=2^{64}$. Hence, $\mathop{\rm{poly}}\nolimits(d)$ sample complexity is unacceptable.

1.2 Our Results

Our first result is an efficient private learner for conjunctions. Our learner is obtained by modifying the strategy outlined above to use the greedy algorithm for set cover in order to choose a small number of literals directly out of the set of all possible $2d$ literals (instead of choosing them out of the set of literals that agree with the positive examples). However, this must be done carefully, as unlike before, we need to ensure that the selected literals will not cause our hypothesis to err on the positive examples. Specifically, in every step of the greedy algorithm we will aim at choosing a literal that eliminates (i.e., evaluates to zero on) a lot of the negative examples without eliminating (essentially) any of the positive examples. In the terminology of the set cover problem, we will have two types of elements – positive and negative elements – and our goal will be to identify a small cover of the negative elements that does not cover (essentially) any of the positive elements. We show,

Theorem 1.2**.**

There exists an efficient $(\varepsilon,\delta)$-differentially private $(\alpha,\beta)$-PAC learner for $\mathop{\rm{CONJ}}\nolimits_{k,d}$ with sample complexity111For simplicity we used the $\tilde{O}$ notation to hide logarithmic factors in $\alpha,\beta,\epsilon,\delta,k$. The dependency in these factors wil be made explicit in the following sections. $\tilde{O}(\frac{1}{\alpha\varepsilon}k\log d)$.

We remark that our techniques extend to disjunctions of literals in a straightforward way, as follows.

Theorem 1.3**.**

There exists an efficient $(\varepsilon,\delta)$-differentially private $(\alpha,\beta)$-PAC learner for the class of all disjunctions (i.e., OR) of at most $k$ literals over $d$ Boolean variables with sample complexity $\tilde{O}(\frac{1}{\alpha\varepsilon}k\log d)$.

We then show that our technique can be used to privately learn (not necessarily convex) polygons in the plane. To see the connection, let us first consider convex polygons, and observe that a convex polygon with $k$ edges can be represented as the intersection of $k$ halfplanes. Thus, as in our learner for Boolean conjunctions, if we could identify a halfplane that eliminates (i.e., evaluates to zero on) a lot of the negative examples without eliminating (essentially) any of the positive examples in the sample, then we could learn convex polygons in iterations. However, recall that in our learner for conjunctions, the parameter $d$ controlled both the description length of examples (since each example specified an assignment to $d$ variables) and the number of possible literals (which was $2d$). Thus, the running time of our algorithm was allowed to be linear in the number of possible literals. The main challenge when applying this technique to (convex) polygons is that the number of possible halfplanes (which will correspond to the parameter $d$) is huge, and our algorithm cannot run in time linear in the number of possible halfplanes.

To recover from this difficulty, we consider the dual plane in which sample points correspond to lines, and show that in that dual plane it is possible to (privately) identify a point (that corresponds to a halfplane in the primal plane) with the required properties (that is, eliminating a lot of negative examples while not eliminating positive examples). The idea is that since there are only $n$ input points, in the dual plane there will be only $n$ lines to consider, which partition the dual plane into at most $n^{2}$ regions. Now, two different halfplanes in the primal plane correspond to two different points in the dual plane, and if these two points fall in the same region, then the two halfplanes behave identically on the input sample. We could therefore partition the halfplanes (in the primal plane) into at most $n^{2}$ equivalence classes w.r.t. the way they behave on the input sample. This fact can be leveraged in order to efficiently implement the algorithm.

Our techniques extend to non-convex polygons, which unlike convex polygons cannot be represented as intersection of halfplanes. It it well known that every (simple222A simple polygon is one which does not intersect itself.) polygon with $k$ edges can be represented as the union of at most $k$ triangles, each of which can be represented as the intersection of at most 3 halfplanes (as a triangle is a convex polygon with 3 edges). In other words, a (simple) polygon with $k$ edges can be represented as a DNF formula (i.e., disjunction of conjunctive clauses) in which each clause has at most 3 literals. As we will see, our techniques can be extended to capture this case efficiently. Our main theorem is the following.

Theorem 1.4** (informal).**

There exists an efficient $(\varepsilon,\delta)$-differentially private $(\alpha,\beta)$-PAC learner for union of (simple) polygons in the plane with sample complexity $\tilde{O}(\frac{1}{\alpha\varepsilon}k\log d)$, where $k$ is the number of edges in the union of polygons and $\log d$ is the description length of examples.

As the greedy algorithm for set cover has many applications in computational learning theory, we hope that our techniques will continue to find much broader use.

2 Preliminaries

We recall standard definitions from learning theory and differential privacy. In the following $X$ is some arbitrary domain. A concept (similarly, hypothesis) over domain $X$ is a predicate defined over $X$. A concept class (similarly, hypothesis class) is a set of concepts.

Definition 2.1** (Generalization Error).**

Let $\mathcal{D}\in\Delta(X)$ be a probability distribution over $X$ and let $c:x\rightarrow\{0,1\}$ be a concept. The generalization error of a hypothesis $h:X\rightarrow\{0,1\}$ w.r.t. $\mathcal{D}$ and $c$ is defined as ${\rm error}_{\mathcal{D}}(c,h)=\Pr_{x\sim\mathcal{D}}[h(x)\neq c(x)]$.

We now recall the notion of PAC learning [29]. Let $C$ and $H$ be a concept class and a hypothesis class over a domain $X$, and let $\mathcal{A}:\left(X\times\{0,1\}\right)^{n}\rightarrow H$ be an algorithm that operates on a labeled database and returns a hypothesis from $H$.

Definition 2.2** (PAC Learner [29]).**

Algorithm $\mathcal{A}$ is an $(\alpha,\beta)$-PAC learner for concept class $C$ using hypothesis class $H$ with sample complexity $n$ if for every distribution $\mathcal{D}$ over $X$ and for every fixture of $c\in C$, given a labeled database $S=\left(\left(x_{i},c(x_{i})\right)\right)_{i=1}^{n}$ where each $x_{i}$ is drawn i.i.d. from $\mathcal{D}$, algorithm $\mathcal{A}$ outputs a hypothesis $h\in H$ satisfying

[TABLE]

The probability is taken over the random choice of the examples in $S$ according to $\mathcal{D}$ and the coin tosses of the learner $\mathcal{A}$.

Without privacy considerations, the sample complexity of PAC learning is essentially characterized by a combinatorial quantity called the Vapnik-Chervonenkis (VC) dimension:

Definition 2.3**.**

Fix a concept class $C$ over domain $X$. A set $\{x_{1},\dots,x_{\ell}\}\in X$ is shattered by $C$ if for every labeling $b\in\{0,1\}^{\ell}$, there exists $c\in C$ such that $b_{1}=c(x_{1}),\dots,b_{\ell}=c(x_{\ell})$. The Vapnik-Chervonenkis (VC) dimension of $C$, denoted $\operatorname{\rm VC}(C)$, is the size of the largest set which is shattered by $C$.

The Vapnik-Chervonenkis (VC) dimension is an important combinatorial measure of a concept class. Classical results in statistical learning theory show that the generalization error of a hypothesis $h$ and its empirical error (observed on a large enough sample) are similar.

Definition 2.4** (Empirical Error).**

Let $S=((x_{i},\sigma_{i}))_{i=1}^{n}\in(X\times\{0,1\})^{n}$ be a labeled sample. The empirical error of a hypothesis $h:X\rightarrow\{0,1\}$ w.r.t. $S$ is defined as ${\rm error}_{S}(h)=\frac{1}{n}|\{i:h(x_{i})\neq\sigma_{i}\}|$.

Theorem 2.5** (VC-Dimension Generalization Bound, e.g. [8]).**

Let $\mathcal{D}$ and $C$ be, respectively, a distribution and a concept class over a domain $X$, and let $c\in C$. For a sample $S=((x_{i},c(x_{i})))_{i=1}^{n}$ where $n\geq\frac{64}{\alpha}(\operatorname{\rm VC}(C)\ln(\frac{64}{\alpha})+\ln(\frac{8}{\beta}))$ and the $x_{i}$ are drawn i.i.d. from $\mathcal{D}$, it holds that

[TABLE]

2.1 Conjunctions and Disjunctions

Definition 2.6**.**

Let $H$ be a concept class over a domain $X$, and let $k\in\N$. We use $H^{\vee k}$ to denote the class of all disjunctions (i.e., OR) of at most $k$ concepts from $H$, and similarly, we denote $H^{\wedge k}$ for the class of all conjunctions (i.e., AND) of at most $k$ concepts from $H$.

The following observation is standard (see, e.g., [19]).

Observation 2.7**.**

For every concept class $H$ we have

[TABLE]

Our strategy in the following sections for privately learning a concept class $C$ will be to use a “simpler” concept class $H$ such that for some (hopefully small) $k$ we have $C\subseteq H^{\vee k}$ or $C\subseteq H^{\wedge k}$.

Example 2.8**.**

Let $\mathop{\rm{DISJ}}\nolimits_{k,d}$ denote the class of all disjunctions (i.e., OR) of at most $k$ literals over $d$ Boolean variables, and similarly, let $\mathop{\rm{CONJ}}\nolimits_{k,d}$ denote the class of all conjunctions (i.e., AND) of at most $k$ literals over $d$ Boolean variables. Trivially, $\mathop{\rm{DISJ}}\nolimits_{k,d}=(\mathop{\rm{DISJ}}\nolimits_{1,d})^{\vee k}$, and $\mathop{\rm{CONJ}}\nolimits_{k,d}=(\mathop{\rm{CONJ}}\nolimits_{1,d})^{\wedge k}$.

2.2 Differential privacy

Two databases $S,S^{\prime}$ are called neighboring if they differ on a single entry.

Definition 2.9** (Differential Privacy [16]).**

Let $\mathcal{A}$ be a randomized algorithm whose input is a database. Let $\varepsilon,\delta\geq 0$. Algorithm $\mathcal{A}$ is $(\varepsilon,\delta)$-differentially private if for all neighboring databases $S,S^{\prime}$ and for any event $T$,

[TABLE]

where the probability is taken over the coin tosses of the algorithm $\mathcal{A}$. When $\delta=0$ we omit it and say that $\mathcal{A}$ is $\varepsilon$-differentially private.

Our learning algorithms are designed via repeated applications of differentially private algorithms on a database. Composition theorems for differential privacy show that the price of privacy for multiple (adaptively chosen) interactions degrades gracefully.

Theorem 2.10** (Composition of Differential Privacy [14, 15, 18]).**

Let $0<\varepsilon,\delta^{\prime}<1$ and $\delta\in[0,1]$. Suppose an algorithm $\mathcal{A}$ accesses its input database $S$ only through $m$ adaptively chosen executions of $(\varepsilon,\delta)$-differentially private algorithms. Then $\mathcal{A}$ is

$(m\varepsilon,m\delta)$-differentially private, and 2. 2.

$(\varepsilon^{\prime},m\delta+\delta^{\prime})$-differentially private for $\varepsilon=\sqrt{2m\ln(1/\delta^{\prime})}\cdot\varepsilon+2m\varepsilon^{2}$.

The most basic constructions of differentially private algorithms are via the Laplace mechanism as follows.

Definition 2.11** (The Laplace Distribution).**

A random variable has probability distribution $\mathop{\rm{Lap}}\nolimits(b)$ if its probability density function is $f(x)=\frac{1}{2b}\exp(-\frac{|x|}{b})$, where $x\in\R$.

Definition 2.12** (Sensitivity).**

A $f$ mapping databases to the reals has sensitivity $s$ if for every neighboring $S,S$, it holds that $|f(S)-f(S^{\prime})|\leq s$.

Theorem 2.13** (The Laplacian Mechanism [16]).**

Let $f$ be a sensitivity $s$ function. The mechanism $\mathcal{A}$ that on input a database $S$ adds noise with distribution $\mathop{\rm{Lap}}\nolimits(\frac{s}{\varepsilon})$ to the output of $f(S)$ preserves $\varepsilon$-differential privacy. Moreover,

[TABLE]

We next describe the exponential mechanism of McSherry and Talwar [26]. Given a database $S$, the exponential mechanism privately chooses a “good” solution $h$ out of a set of possible solutions $H$ (in our context, $H$ will be a hypothesis class). This “goodness” is quantified using a quality function that matches solutions to scores.

Definition 2.14** (Quality function).**

A quality function $q=q(S,h)$ maps a database $S$ and a solution $h\in H$ to a real number, identified as the score of the solution $h$ w.r.t. the database $S$. We say that $q$ has sensitivity $s$ if $q(\cdot,h)$ has sensitivity $s$ for every $h\in H$.

Given a sensitivity-1 quality function $q$ and a database $S$, the exponential mechanism chooses a solution $h\in H$ with probability proportional to $\exp\left(\epsilon\cdot q(S,h)/2\right)$.

Proposition 2.15** (Properties of the exponential mechanism).**

(i) The exponential mechanism is $\varepsilon$-differentially private. (ii) Let $\lambda>0$. The exponential mechanism outputs a solution $h$ such that $q(S,h)\leq\max_{f\in H}\{q(S,f)\}-\lambda$ with probability at most $|H|\cdot\exp(-\varepsilon\lambda/2)$.

3 A Generic Construction via Set Cover

In this section we present our generic construction for privately learning a concept class $C$ containing concepts that can be written as the conjunction or the disjunction of functions in a (hopefully simpler) class $H$. For readability we focus on conjunctions. The extension to disjunction is straightforward.

Claim 3.1**.**

Fix a target function $c^{*}\in C$, and consider the execution of SetCoverLearner on a sample $S=\{(x_{i},c^{*}(x_{i}))\}_{i=1}^{n}$. Assume that every run of the selection procedure $\mathcal{A}$ in Step 1e returns a hypothesis $h_{j}$ s.t. $q(h_{j})\geq\max_{f\in H}\{q(f)\}-\lambda$. Then, with probability at least $1-\beta$ it holds that $h_{fin}$ errs on at most $\max\left\{\frac{\alpha n}{2},\;\frac{8k}{\varepsilon}\log\left(\frac{2}{\alpha}\right)\log\left(\frac{2k}{\beta}\log\frac{2}{\alpha}\right)\right\}+4k\lambda\log\frac{2}{\alpha}$ example in $S$.

Proof.

First observe that there are $2k\log\frac{2}{\alpha}$ draws from $\mathop{\rm{Lap}}\nolimits\left(\frac{2k}{\varepsilon}\log\frac{2}{\alpha}\right)$ throughout the execution. By the properties of the Laplace distribution, with probability at least $1-\beta$ it holds that the maximum absolute value of these random variables is at most $\Delta=\frac{2k}{\varepsilon}\log\left(\frac{2}{\alpha}\right)\log\left(\frac{2k}{\beta}\log\frac{2}{\alpha}\right)$. We continue with the analysis assuming that this is the case. In particular, this means that in every iteration $j$ we have $|S^{0}|-2\Delta\leq b_{j}\leq|S^{0}|$. Thus, in every iteration there exists a hypothesis $\tilde{h}\in H$ with $q(\tilde{h})\geq 0$. To see this, recall that the target concept $c^{*}$ can be written as $c^{*}=h^{*}_{1}\wedge\dots\wedge h^{*}_{k}$ for $h^{*}_{1},\dots,h^{*}_{k}\in H$. Hence, in every iteration $j$ there is a hypothesis $\tilde{h}\in H$ that correctly classifies all of the (remaining) positive points in $S$ while correctly classifying at least $1/k$ fraction of the (remaining) negative points in $S$, i.e., at least $|S^{0}|/k\geq b_{j}/k$ negative points. Such a hypothesis $\tilde{h}$ satisfies $q(\tilde{h})=0$. By our assumption on the selection procedure $\mathcal{A}$, we therefore have that in each iteration $j$, the selection procedure identifies a hypothesis $h_{j}$ s.t. $q(h_{j})\geq-\lambda$.

By the definition of $q$, in every iteration $j$ we have that the selected $h_{j}$ misclassifies at most $\lambda$ of the remaining positive examples in $S$. Therefore, $h_{fin}$ misclassifies at most $2k\lambda\log\frac{2}{\alpha}$ positive examples in $S$. Moreover, in every iteration $j$ s.t. $|S^{0}|\geq 2k\lambda+4\Delta$ we have that $h_{j}$ classifies correctly at least $\frac{1}{2k}$ fraction of the negative examples in $S$. To see this, observe that as $q(h_{j})\geq-\lambda$ we have

[TABLE]

That is, either there exists an iteration $j$ in which number of negative points in $S$ drops below $2k\lambda+4\Delta$, or every iteration shrinks the number of negative examples by a factor of $\frac{1}{2k}$, in which case after $2k\log\frac{2}{\alpha}$ iterations there could be at most $\frac{\alpha n}{2}$ negative points in $S$. Observe that $h_{fin}$ does not err on negative points that were removed from $S$, and therefore, there could be at most $\max\left\{\frac{\alpha n}{2},\;2k\lambda+4\Delta\right\}$ negative points on which $h_{fin}$ errs. Overall, $h_{fin}$ errs on at most $\max\left\{\frac{\alpha n}{2},\;4\Delta\right\}+4k\lambda\log\frac{2}{\alpha}$ points in $S$. ∎

Claim 3.1 ensures that if at every step $\mathcal{A}$ picks $h_{j}$ of high quality, then (w.h.p.) algorithm SetCoverLearner returns a hypothesis from $H^{\wedge 2k\log\frac{2}{\alpha}}$ with low empirical error. Combining this with standard generalization bounds and with Observation 2.7 (that bounds the VC dimension of $H^{\wedge 2k\log\frac{2}{\alpha}}$) we get the following theorem.

Theorem 3.2**.**

Let $C,H,k$ be two concept classes and an integer such that $C\subseteq H^{\wedge k}$. Let $\mathcal{A}$ be a selection procedure that takes a database $S$ and a quality function $q$, and returns a hypothesis $h\in H$ such that $q(h_{j})\geq\max_{f\in H}\{q(f)\}-\lambda$ with probability at least $1-\frac{\beta}{4k\log(2/\alpha)}$. Then, algorithm SetCoverLearner with $\mathcal{A}$ as the selection procedure is an $(\alpha,\beta)$-PAC learner for $C$ with sample complexity

[TABLE]

3.1 Tuning the selection procedure

If $H$ is finite, then one could directly implement the selection procedure $\mathcal{A}$ using the exponential mechanism of McSherry and Talwar [26] to find a hypothesis $h_{j}$ with large $q(h_{j})$ at each iteration. In order to guarantee that all of the $\approx k$ iterations of algorithm SetCoverLearner satisfy together $(\varepsilon,\delta)$-differential privacy, it suffices that each application of the exponential mechanism satisfies $\hat{\varepsilon}\approx\frac{\varepsilon}{\sqrt{k}}$-differential privacy (see Theorem 2.10). When choosing such an $\hat{\varepsilon}$, the exponential mechanism identifies, in every iteration, an $h_{j}$ such that

[TABLE]

This gives a selection procedure $\mathcal{A}$ which selects $h_{j}$ with $q(h_{j})\geq\max_{f\in H}\{q(f)\}-\lambda$, for $\lambda\approx\frac{\sqrt{k}}{\varepsilon}\log|H|$.

Example 3.3**.**

There exist efficient $(\varepsilon,\delta)$-differentially private $(\alpha,\beta)$-PAC learners for $\mathop{\rm{CONJ}}\nolimits_{k,d}$ and for $\mathop{\rm{DISJ}}\nolimits_{k,d}$ with sample complexity $n=\tilde{\Theta}\left(\frac{1}{\alpha\varepsilon}\cdot k^{1.5}\log d\right)$.

The reason for the dependency in $k^{1.5}$ in the above example, is that for the privacy analysis we wanted to ensure that each iteration was differentially private with parameter $\approx\varepsilon/\sqrt{k}$ (because when composing $\ell$ differentially private mechanisms the privacy budget deteriorates proportionally to $\sqrt{\ell}$). This resulted in $\approx\sqrt{k}/\varepsilon$ misclassified points per iteration (and there are $\approx k$ iterations). As we next explain, in our case, the privacy parameter does not need to deteriorate with the number of iterations, which allows us to improve the sample complexity by a $\sqrt{k}$ factor. Our approach to proving this improved bound builds on the analysis of Gupta et al. [22] for their private algorithm for set cover. The main difference is that we have both positive and negative examples, which we need to handle differently. As we next explain, this will be achieved using Step 1c of SetCoverLearner.

Claim 3.4**.**

Let $\varepsilon\in(0,1)$ and $\delta<1/e$. Instantiating SetCoverLearner with the exponential mechanism as the selection procedure $\mathcal{A}$ with privacy parameter $\hat{\varepsilon}=\frac{\varepsilon}{2\ln(e/\delta)}$ (for each iteration) satisfies $(\varepsilon,\delta)$-differential privacy.

We first present an intuitive (and oversimplified) overview of the proof. Consider two neighboring databases $S$ and $S^{\prime}$ such that $S^{\prime}=S\cup\{(x^{*},\sigma^{*})\}$, and let us focus this intuitive overview on the case where $\sigma^{*}=0$. Fix a possible output $\vec{h}=(h_{1},h_{2},\dots,h_{2k\log\frac{2}{\alpha}})$ of SetCoverLearner. We will analyze the ratio

[TABLE]

Let $t$ be such that $h_{t}$ is the first hypothesis in this output vector satisfying $h_{t}(x^{*})=0$. Observe that after the $t$th iteration, the executions on $S$ and on $S^{\prime}$ continue exactly the same, since $(x^{*},\sigma^{*})$ is removed from $S^{\prime}$ during the $t$th iteration (because in every iteration we remove all input elements on which the selected hypothesis evaluates to 0). Intuitively, if $t$ is small then we only need to pay (in the privacy analysis) for a small number of iterations. In general, however, $t$ might be as large as $2k\log\frac{2}{\alpha}$, and accounting for that many iterations in the privacy analysis is exactly what we are trying to avoid.

Recall that each iteration $j$ of SetCoverLearner draws a random noise $w_{j}$ from $\left\lfloor\mathop{\rm{Lap}}\nolimits\left(\frac{2k}{\varepsilon}\log\frac{2}{\alpha}\right)\right\rfloor$. Let us denote these noises as they are in the execution on $S$ as $\vec{w}=(w_{1},\dots,w_{2k\log\frac{2}{\alpha}})$ and in the execution on $S^{\prime}$ as $\vec{w}^{\prime}=(w^{\prime}_{1},\dots,w^{\prime}_{2k\log\frac{2}{\alpha}})$. Furthermore, let us assume that $w^{\prime}_{j}=w_{j}-1$ for every $j\leq t$ and that $w^{\prime}_{j}=w_{j}$ for every $j>t$. By the properties of the Laplace distribution, this assumption distorts our bound on the ratio in expression (1) by at most an $e^{\varepsilon}$ factor. (In a sense, for these random noises we do account for all $2k\log\frac{2}{\alpha}$ potential iterations by sampling random noises with larger variance. However, this larger variance is mitigated by the fact that in the quality function $q$ we divide noises by $k$, and hence, we do not incurr an increase of $\mathop{\rm{poly}}\nolimits(k)$ in the sample complexity due to this issue.)

We have already established that after the $t$th iteration, the two executions are identical. In addition, due to our assumption on $\vec{w}$ and $\vec{w}^{\prime}$, during the first $t$ iterations, the only hypotheses with different qualities (between the execution on $S$ and on $S^{\prime}$) or those hypotheses that label $x^{*}$ as 0. This is because if a hypothesis $h$ labels $x^{*}$ as 1, then $(x^{*},0)$ only effects the quality $q(h)$ via the noisy estimation for the size of $S^{0}$ (denoted as $b_{j}$ in the algorithm), which by our assumption on $\vec{w}$ and $\vec{w}^{\prime}$ is the same in the two executions (because the difference in the noise cancels out the difference in $|S^{0}|$). To summarize, after conditioning on $\vec{w}$ and $\vec{w}^{\prime}$, the additional example $(x^{*},0)$ causes the two executions to differ only in their first $t$ iterations, and within these $t$ iterations it affects only the qualities of the hypotheses that label $x^{*}$ as zero. This can be formalized to bound the ratio in expression (1) by $\lesssim\prod_{j=1}^{t}\exp\left(\varepsilon\cdot p_{j}\right)$, where $p_{j}$ is the probability that a hypothesis that labels $x^{*}$ as 0 is chosen at step $j$ of the algorithm. The proof then concludes by arguing that if these probabilities $\{p_{j}\}$ are small then they reduce our privacy costs (since they multiply $\varepsilon$), and if these probabilities $\{p_{j}\}$ are large then the index $t$ should be small (since we are likely to identify a hypothesis that labels $x^{*}$ as zero quickly, and $t$ is the index of the first such hypothesis), and therefore we must only account for the privacy loss incurred during a small number of iterations. We now proceed with the formal proof.

Proof of Claim 3.4.

Let $S$ and $S^{\prime}$ be two neighboring databases such that $S\triangle S^{\prime}=\{(x^{*},\sigma^{*})\}$. Fix a possible output of SetCoverLearner $\vec{h}=(h_{1},h_{2},\dots,h_{2k\log\frac{2}{\alpha}})$, and let $q_{j,S,w_{j}}(h)$ denote the quality $q(h)$ of a hypothesis $h\in H$ during the $j$th iteration of the algorithm when running on $S$, conditioned on $h_{1},\dots,h_{j-1}$ being chosen in the previous steps and on the value of $w_{j}$. Let $t$ be such that $h_{t}$ is the first hypothesis in $\vec{h}$ satisfying $h_{t}(x^{*})=0$.

Case (a):

$S^{\prime}=S\cup\{(x^{*},\sigma^{*})\}$ and $\sigma^{*}=1$. Fix a noise vector $\vec{w}$. We can calculate

[TABLE]

After $t$, the remaining elements in $S$ and $S^{\prime}$ are identical, and all subsequent terms cancel. Moreover, except for the $t$th term, the numerators of both the top and the bottom expressions cancel, since all the relevant scores are equal.

We are currently assuming that $S^{\prime}=S\cup\{(x^{*},\sigma^{*})\}$ and $\sigma^{*}=1$. Hence, for every hypothesis $f$ and every step $j$ we have that $q_{j,S,w_{j}}(f)-1\leq q_{j,S^{\prime},w_{j}}(f)\leq q_{j,S,w_{j}}(f)$, since adding a positive example to the database can decrease the quality by at most 1 (and it cannot increase the quality). Hence, the first term above is at most $\exp(\hat{\varepsilon})$, and the second term is at most 1. As this holds for every possible value of the noise vector $\vec{w}$, we get that

[TABLE]

Case (b):

$S^{\prime}=S\cup\{(x^{*},\sigma^{*})\}$ and $\sigma^{*}=0$. Fix a noise vector $\vec{w}$, and let $\vec{w}^{\prime}$ be such that $w^{\prime}_{j}=w_{j}-1$ for every $j\leq t$ and $w^{\prime}_{j}=w_{j}$ for every $j>t$. (Recall that $t$ is the index of the first hypothesis in the output vector $\vec{h}$ that labels $x^{*}$ as 0.) We have that

[TABLE]

As before, after $t$ the remaining elements in $S$ and $S^{\prime}$ are identical, and all subsequent terms cancel (recall that $w_{j}=w^{\prime}_{j}$ for every $j>t$). In addition, due to our choice of $w^{\prime}_{j}=w_{j}-1$ for every $j\leq t$, we again get that, except for the $t$th term, the numerators of both the top and the bottom expression cancel, since all the relevant scores are equal.

We are currently analyzing the case where $S^{\prime}=S\cup\{(x^{*},\sigma^{*})\}$ and $\sigma^{*}=0$. Hence, the first term above is $\exp(-\hat{\varepsilon})<1$, because $q_{t,S^{\prime},w^{\prime}_{t}}(h_{t})=q_{t,S,w_{t}}(h_{t})+1$. Moreover, for every $j\leq t$ we have that $w^{\prime}_{j}=w_{j}-1$. Hence, for every $j\leq t$ and every hypothesis $f$ s.t. $f(x^{*})=1$ we have $q_{j,S^{\prime},w^{\prime}_{t}}(f)=q_{j,S,w_{t}}(f)$. Also, for every $j\leq t$ and every hypothesis $f$ s.t. $f(x^{*})=0$ we have $q_{j,S^{\prime},w^{\prime}_{t}}(f)=q_{j,S,w_{t}}(f)+1$. Therefore we have

[TABLE]

where $p_{j}(S,w_{j})$ is the probability that a hypothesis that labels $x^{*}$ as 0 is chosen at step $j$ of the algorithm running on $S$, conditioned on picking the hypotheses $h_{1},\dots,h_{j-1}$ in the previous steps, and on the noise $w_{j}$.

For an instance $S$ and an example $x^{*}$, we say that an output $\vec{h}=(h_{1},\dots,h_{2k\log\frac{2}{\alpha}})$ is $\lambda$-bad if $\sum_{j=1}^{2k\log\frac{2}{\alpha}}p_{j}(S,w_{j})\cdot\mathbbm{1}{\{h_{1}(x^{*})=h_{2}(x^{*})=\dots=h_{j}(x^{*})=1\}}>\lambda$, where $p_{j}(S,w_{j})$ is as defined above. We call the output $\vec{h}$ $\lambda$-good otherwise. We first consider the case when the output $\vec{h}$ is $\ln(1/\delta)$-good. By the definition of $t$ we have

[TABLE]

Then we can bound the expression (2) by

[TABLE]

So, for every $\ln(1/\delta)$-good output $\vec{h}$ for $S$ we have

[TABLE]

As for a $\ln(1/\delta)$-bad output, the following lemma shows the probability that $\texttt{SetCoverLearner}(S)$ outputs a $\ln(1/\delta)$-bad output (for $S$) is at most $\delta$.

Lemma 3.5** ([22]).**

Consider the following $n$ round probabilistic process. In each round, an adversary chooses a $p_{j}\in[0,1]$ possibly based on the first $(j-1)$ rounds and a coin is tossed with heads probability $p_{j}$. Let $Z_{j}$ be the indicator for the the event that no coin comes up heads in the first $j$ steps. Let $Y$ denote the random variable $\sum_{j=1}^{n}p_{j}Z_{j}$. Then for any $y$ we have $\Pr[Y>y]\leq\exp(-y)$.

Specifically, to map our setting to that of Lemma 3.5, consider running SetCoverLearner as follows. When choosing a hypothesis $h_{j}$ in step $j$, the algorithm first tosses a coin whose heads probability is $p_{j}(S,w_{j})$ to decide whether to pick a hypothesis that labels $x^{*}$ as 0 or not. Then it uses a second source of randomness to determine the hypothesis $h_{j}$ itself, sampling with the appropriate conditional probabilities based on the outcome of the coin.

Thus, for any set $F$ of outcomes, we have

[TABLE]

A similar analysis holds for the case where $S=S^{\prime}\cup\{(x^{*},\sigma^{*})\}$. ∎

For example, by combining Claim 3.4 with Claim 3.1, we get improved learners for conjunctions and disjunctions:

Theorem 3.6**.**

There exist efficient $(\varepsilon,\delta)$-differentially private $(\alpha,\beta)$-PAC learners for $\mathop{\rm{CONJ}}\nolimits_{k,d}$ and $\mathop{\rm{DISJ}}\nolimits_{k,d}$ with sample complexity $n=\tilde{O}\left(\frac{1}{\alpha\varepsilon}\cdot k\log d\right)$.

4 Convex Polygons in the Plane

In this section we show how our generic construction from the previous section applies to convex polygons in a (discrete version of the) Euclidean plane. This is an important step towards our construction for (not necessarily convex) polygons.

We represent a convex polygon with $k$ edges as the intersection of $k$ halfplanes. A halfplane over 2 can be represented using 3 parameters $a,b,c\in\R$ with $f_{a,b,c}(x,y)=1$ iff $cy\geq ax+b$. Denote the set of all such halfplanes over 2 as

[TABLE]

We can now define the class of convex polygons with $k$ edges over 2 as

[TABLE]

For a parameter $d\in\N$, let $X_{d}=\{0,1,2,\dots,d\}$, and let $X_{d}^{2}=(X_{d})^{2}$ denote a discretization of the Euclidean plane, in which each axis consists of the points in $X_{d}$. We assume that our examples are from $X_{d}^{2}$. Hence, as explained next, we are able to represent a halfplane using only two real parameters $a,b\in\R$ and a bit $z\in\{\pm 1\}$. The parameters $a$ and $b$ define the line $y=ax+b$, and the parameter $z$ determines whether the halfplane is “above” or “below” the line. In other words, $f_{a,b,z}(x,y)=1$ iff $zy\geq z(ax+b)$. Even though in this representation we do not capture vertical lines, for our purposes, vertical lines will not be needed. The reason is that when the examples come from the discretization $X_{d}^{2}$, a vertical line can always be replaced with a non-vertical line such that the corresponding halfplanes behave exactly the same on all of $X_{d}^{2}$. Moreover, since the discretization $X^{2}_{d}$ is finite, it suffices to consider bounded real valued parameters $a,b\in[-2d^{2},2d^{2}]$ (see Observation 4.2 below). Actually, by letting $a$ reside in a bigger range, we can encode the bit $z$ in $a$, and represent a halfplane using only two real numbers. We denote the set of all such halfplanes as

[TABLE]

Observation 4.1**.**

Let $a,b\in[-2d^{2},2d^{2}]$ and $z\in\{\pm 1\}$, and define $f_{a,b,z}(x,y)=1$ iff $zy\geq z(ax+b)$. Then, there exists an $\hat{f}\in\operatorname*{\tt HALFPLANE}\nolimits_{d}$ such that $\hat{f}\equiv f_{a,b,z}(x,y)$.

Proof sketch.

If $z=1$ then define $\hat{a}=a$. Otherwise, if $z=-1$ then define $\hat{a}=a+4d^{2}$. Observe that in both cases $f_{\hat{a},b}\in\operatorname*{\tt HALFPLANE}\nolimits_{d}$ is equivalent to $f_{a,b,z}$. ∎

Observation 4.2**.**

For every $f\in\operatorname*{\tt HALFPLANE}\nolimits$ there exists an $\hat{f}\in\operatorname*{\tt HALFPLANE}\nolimits_{d}$ such that for every $(x,y)\in X^{2}_{d}$ we have $f(x,y)=\hat{f}(x,y)$.

Proof sketch.

Let $f\in\operatorname*{\tt HALFPLANE}\nolimits$. By Observation 4.1, it suffices to show that there exists a halfplane $\hat{f}_{a,b,z}$ equivalent to $f$ of the form $\hat{f}_{a,b,z}(x,y)=1$ iff $zy\geq z(ax+b)$, where $a,b\in[-2d^{2},2d^{2}]$. Without loss of generality, we may assume that $f$ “touches” two points $(x_{1},y_{1}),(x_{2},y_{2})\in X^{2}_{d}$, as otherwise we could “tilt” $f$ to make it so without effecting the way it labels points in $X^{2}_{d}$. Hence, $f$ can be defined by the line equation $(y-y_{1})(x_{2}-x_{1})=(y_{2}-y_{1})(x-x_{1})$, together with a bit $z\in\{\pm 1\}$ that determines whether the halfplane is “above” or “below” that line. First observe that if $x_{1}\neq x_{2}$, then this line equation can be rewritten as

[TABLE]

where $a,b\in[-d^{2},d^{2}]$ because $x_{1},y_{1},x_{2},y_{2}\in X_{d}$ and $x_{1}\neq x_{2}$. That is, the halfplane $f$ can be defined as $f(x,y)=1$ iff $zy\geq z(ax+b)$, as required. Next note that if $x_{1}=x_{2}$, then $f$ is described by the vertical line $x=x_{1}$ and a bit $z\in\{\pm 1\}$, where $f(x,y)=1$ iff $zx\geq zx_{1}$. Now consider the line that passes through $(x_{1},0)$ and $(x_{1}+0.5,d)$, and a line that passes through $(x_{1},0)$ and $(x_{1}-0.5,d)$. One of these two lines, depending on $z$, defines a halfplane that splits $X_{d}^{2}$ identically to $f$. Such a line can be described as $zy=z(ax+b)$ for $a,b\in[-2d^{2},2d^{2}]$. ∎

Remark 4.3**.**

We think of the discretization size $d$ as a large number, e.g., $d=2^{64}$. The runtime and the sample complexity of our algorithms is at most logarithmic in $d$.

A consequence of Observation 4.2 is that, in order to learn $\operatorname*{\tt CONVEX-k-GON}\nolimits$ over examples in $X^{2}_{d}$, it suffices to describe a learner for the class $\operatorname*{\tt HALFPLANE}\nolimits_{d}^{\wedge k}$ over examples in $X^{2}_{d}$. As we next explain, this can be done using our techniques from Section 3. Concretely, we need to specify the selection procedure used in Step 1e of algorithm SetCoverLearner, for privately choosing a hypothesis from $\operatorname*{\tt HALFPLANE}\nolimits_{d}$. Our selection procedure appears in algorithm SelectHalfplane.

Privacy analysis of SelectHalfplane.

Consider running algorithm SelectHalfplane with a score function $q$ whose sensitivity is (at most) 1, and observe that, as in the standard analysis of the exponential mechanism [26], algorithm SelectHalfplane satisfies $2\varepsilon$-differential privacy. To see this, fix two neighboring databases $S,S^{\prime}$, and denote the probability density functions in the execution on $S$ and on $S^{\prime}$ as $p_{S}(\hat{a},b)$ and $p_{S^{\prime}}(\hat{a},b)$, respectively. Since $q$ is of sensitivity 1, for every $(\hat{a},b)\in[-2d^{2},6d^{2}]\times[-2d^{2},2d^{2}]$ we have that $p_{S}(\hat{a},b)\leq e^{2\varepsilon}p_{S^{\prime}}(\hat{a},b)$. Hence, for any set of possible outcomes $F$ we have $\Pr[\texttt{SelectHalfplane}(S)\in F]\leq e^{2\varepsilon}\cdot\Pr[\texttt{SelectHalfplane}(S^{\prime})\in F]$, as required. Moreover, a similar analysis to that of Claim 3.4 shows the following.

Claim 4.4**.**

When instantiating algorithm SetCoverLearner with SelectHalfplane as the selection procedure, in order for the whole execution to satisfy $(\varepsilon,\delta)$-differential privacy, it suffices to execute each instance of SelectHalfplane with a privacy parameter $\hat{\varepsilon}=O\left(\varepsilon/\log(1/\delta)\right)$.

4.1 Utility analysis of SelectHalfplane

In algorithm SelectHalfplane we identify points in $X^{2}_{d}$ with lines in $D^{2}$ and vice verse. The following observation states that if two points in $D^{2}$ belong to the same region (as defined in Step 3) then these two points correspond to halfplanes in $X^{2}_{d}$ that agree on every point in the input sample $S$. This allows us to partition the halfplanes (in the primal plane) into a small number of equivalence classes.

Observation 4.5**.**

Consider the execution of SelectHalfplane on a sample $S$, and let $\hat{R}=\{r^{j}_{i}\}$ be the regions defined in Step 3 (for $j\in\{1,2\}$). For every region $r^{j}_{i}\in\hat{R}$, for every two points in this region $(a_{1},b_{1}),(a_{2},b_{2})\in r^{j}_{i}$, and for every example $(x,y)$ in the sample $S$ we have $f_{a_{1},b_{1}}(x,y)=f_{a_{2},b_{2}}(x,y)$.

Proof.

Fix two points $(a_{1},b_{1}),(a_{2},b_{2})$ that belong to the same region in $\hat{R}$. By the definition of the regions in $\hat{R}$, for every example $(x,y)$ in the sample $S$ we have that

[TABLE]

and hence, $f_{a_{1},b_{1}}(x,y)=1$ iff $f_{a_{2},b_{2}}(x,y)=1$. ∎

In particular, Observation 4.5 shows that the function $p$ defined in Step 6 indeed defines a probability density function, as for $F=[-2d^{2},6d^{2}]$ and $D=[-2d^{2},2d^{2}]$ we have

[TABLE]

We also need to argue about the area of the region in the dual plane that corresponds to hypotheses with high quality (as the probability of a choosing a hypotheses from that region is proportional to its area). This is done in the following claim.

Claim 4.6**.**

Consider the execution of SelectHalfplane on a sample $S$, and let $w_{1},\dots,w_{|R|}$ denote the areas of the regions defined in Step 3. Then for every $i$ we have that $w_{i}\geq d^{-4}/4$.

Proof.

We will show that every two different vertices of the regions in $R$ are at distance at least $1/d^{2}$, and hence, the minimal possible area is that of a equilateral triangle with edge length $1/d^{2}$, which has area $\frac{\sqrt{3}}{4d^{4}}$.

To show this lower bound on the distance between a pair of vertices, let $\ell_{x_{1},y_{1}},\ell_{x_{2},y_{2}},\ell_{x_{3},y_{3}},\ell_{x_{4},y_{4}}$ be 4 lines in $S_{\rm dual}$, and assume that $\ell_{x_{1},y_{1}}$ and $\ell_{x_{2},y_{2}}$ intersect at $(a_{1,2},b_{1,2})$, and that $\ell_{x_{3},y_{3}}$ and $\ell_{x_{4},y_{4}}$ intersect at $(a_{3,4},b_{3,4})$. Moreover, assume that these two intersection points are different. We can write the coordinates of these intersection points as

[TABLE]

[TABLE]

Now if $a_{1,2}\neq a_{3,4}$, then

[TABLE]

and if $a_{1,2}=a_{3,4}$, then

[TABLE]

∎

The following lemma states the utility guarantees of SelectHalfplane.

Lemma 4.7**.**

Consider the execution of SelectHalfplane on a sample $S$, and assume that there exists a hypothesis $f\in\operatorname*{\tt HALFPLANE}\nolimits_{d}$ with $q(f)\geq\lambda$. Then the probability that SelectHalfplane outputs a hypothesis $f^{\prime}$ with $q(f^{\prime})<\lambda-\frac{8}{\varepsilon}\ln(\frac{2d}{\beta})$ is at most $\beta$.

Proof.

Denote $F=[-2d^{2},6d^{2}]$ and $D=[-2d^{2},2d^{2}]$. Let $\hat{R}=\{r^{1}_{1},r^{2}_{1},\dots,r^{1}_{|R|},r^{2}_{|R|}\}$ denote the regions defined in Step 3, and let $B\subseteq\hat{R}$ denote the subset of all regions s.t. the halfplanes that correspond to points in these regions have quality less than $\lambda-\frac{8}{\varepsilon}\ln(\frac{2d}{\beta})$. Then the probability that SelectHalfplane outputs a hypothesis $f^{\prime}$ with $q(f^{\prime})<\lambda-\frac{8}{\varepsilon}\ln(\frac{2d}{\beta})$ is at most

[TABLE]

∎

Combining Lemma 4.7 with Claims 3.1 and 4.4 yields our private learners for convex polygons:

Theorem 4.8**.**

There exists an efficient $(\varepsilon,\delta)$-differentially private $(\alpha,\beta)$-PAC learner for $\operatorname*{\tt CONVEX-k-GON}\nolimits$ over examples from $X_{d}^{2}$ with sample complexity

[TABLE]

5 Extension to Union of Non-Convex Polygons

In this section we briefly describe how our techniques from the previous sections can be used to learn the class of (simple) polygons in the plane, as defined next. For a simple and closed curve333A curve is simple and closed if it does not cross itself and ends at the same point where it begins. $C$, we use $\operatorname*{\tt interior}(C)$ to denote the union of $C$ and its bounded area. We define the class of all polygons in the plane with (at most) $k$ edges as

[TABLE]

By standard arguments in computational geometry, every such polygon with $k$ edges can be represented as the union of at most $k$ triangles, each of which can be represented as the intersection of at most 3 halfplanes (since a triangle is a convex polygon with 3 edges). Let us denote the class of all triangles in the plane as $\operatorname*{\tt TRIANGLE}\nolimits$. Hence,

[TABLE]

Thus, in order to learn polygons with $k$ edges, it suffices to construct a learner for the class $\left(\operatorname*{\tt HALFPLANE}\nolimits^{\wedge 3}\right)^{\vee k}$. In fact, this class captures unions of polygons with a total of at most $k$ edges. In addition, similar arguments to those given in Section 4 show that if input examples come from $X_{d}^{2}=\{0,1,\dots,d\}^{2}$, then it suffices to construct a learner for $\left(\operatorname*{\tt HALFPLANE}\nolimits_{d}^{\wedge 3}\right)^{\vee k}$, which we can do using our techniques from Sections 3 and 4.

First, as we mentioned, a straightforward modification to algorithm SetCoverLearner yields an algorithm for learning classes of the form $C\subseteq H^{\vee k}$ (instead of $C\subseteq H^{\wedge k}$ as stated in Section 3). Now, to get an efficient construction, we need to specify the selection procedure for choosing a hypothesis $h_{j}\in\operatorname*{\tt HALFPLANE}\nolimits_{d}^{\wedge 3}$ in each step of SetCoverLearner. As before, given an input sample $S$ we consider the dual plane $D^{2}$ s.t. every input example in $S$ from the primal plane corresponds to a line in the dual plane, and every point from the dual plane corresponds to a halfplane in the primal plane. Recall that in the previous section we identified a hypothesis (which was a halfplane) with a point in the dual plane. The modification is that now a hypothesis is a triangle which we identify with three points in the dual plane (these 3 points correspond to 3 halfplanes in the primal plane, whose intersection is a triangle). Our modified selection procedure is presented as algorithm SelectTriangle.

We use $\operatorname*{\tt k-UNION-GON}\nolimits$ to denote the class of all unions of (simple) polygons with a total of at most $k$ edges. That is, every hypothesis $h\in\operatorname*{\tt k-UNION-GON}\nolimits$ can be written as $h=h_{1}\vee\dots\vee h_{m}$ for $(h_{1},\dots,h_{m})\in\left(\operatorname*{\tt k_{1}-GON}\nolimits\times\dots\times\operatorname*{\tt k_{m}-GON}\nolimits\right)$ where $k_{1}+\dots+k_{m}\leq k$. A similar analysis to that of the previous section shows the following result.

Theorem 5.1**.**

There exists an efficient $(\varepsilon,\delta)$-differentially private $(\alpha,\beta)$-PAC learner for $\operatorname*{\tt k-UNION-GON}\nolimits$ over examples from $X_{d}^{2}$ with sample complexity

[TABLE]

6 Conclusion and Future Work

In this work we presented a computationally efficient differentially private PAC learner for simple geometric concepts in the plane, which can be described as the union of polygons. Our results extend to higher dimensions by replacing lines with hyperplanes, and triangles with simplices. The running time, however, depends exponentially on the dimension. Our results also extend, via linearization, to other simple geometric concepts whose boundaries are defined by low degree polynomials, such as balls. In general, the dimension of the linearization depends on the degrees of the polynomials. This motivates the open problem of improving the dependency of the running time on the dimension of the problem.