Private Hypothesis Selection

TL;DR

This paper introduces a differentially private hypothesis selection algorithm that efficiently identifies a distribution close to the unknown target, with optimal sample complexity for various distribution classes, enabling private distribution learning.

Contribution

It presents a novel differentially private hypothesis selection method with minimal sample complexity, applicable to infinite classes and enabling private distribution learning.

Findings

Achieves near-optimal sample complexity for private hypothesis selection.

Handles infinite hypothesis classes with $( ext{ε,δ})$-differential privacy.

Enables private distribution-free PAC learning.

Abstract

We provide a differentially private algorithm for hypothesis selection. Given samples from an unknown probability distribution $P$ and a set of $m$ probability distributions $\mathcal{H}$, the goal is to output, in a $\varepsilon$-differentially private manner, a distribution from $\mathcal{H}$ whose total variation distance to $P$ is comparable to that of the best such distribution (which we denote by $\alpha$). The sample complexity of our basic algorithm is $O\left(\frac{\log m}{\alpha^2} + \frac{\log m}{\alpha \varepsilon}\right)$, representing a minimal cost for privacy when compared to the non-private algorithm. We also can handle infinite hypothesis classes $\mathcal{H}$ by relaxing to $(\varepsilon,\delta)$-differential privacy. We apply our hypothesis selection algorithm to give learning algorithms for a number of natural distribution classes, including Gaussians, product distributions, sums of independent random variables, piecewise polynomials, and mixture classes. Our hypothesis selection procedure allows us to generically convert a cover for a class to a learning algorithm, complementing known learning lower bounds which are in terms of the size of the packing number of the class. As the covering and packing numbers are often closely related, for constant $\alpha$, our algorithms achieve the optimal sample complexity for many classes of interest. Finally, we describe an application to private distribution-free PAC learning.

Equations

$$n=\Omega \left(\frac{\log |{\mathcal{C}}_{\alpha }|}{{\alpha }^2}+\frac{\log |{\mathcal{C}}_{\alpha }|}{\alpha \epsilon }\right).$$

$$d_{\mathrm{TV}}(P,Q)=\underset{S\subseteq \Omega }{\max }P(S)-Q(S)=\frac{1}{2}{\int }_{x\in \Omega }|P(x)-Q(x)|dx=\frac{1}{2}\Vert P-Q{\Vert }_1\in [0,1].$$

$$\mathrm{Pr}\left[T(D)\in S\right]\le e^{\epsilon }\mathrm{Pr}[T(D^{\prime })\in S]+\delta .$$

$$R_{\alpha }(M(D)||M(D^{\prime }))\le \rho \alpha ,$$

$$\Delta (q)=\underset{r\in \mathcal{R},D\sim D^{\prime }}{\max }\left|q(D,r)-q(D^{\prime },r)\right|.$$

$$q(D,r)\ge \underset{r^{\prime }\in \mathcal{R}}{\max }q(D,r^{\prime })-\frac{2\Delta (q)\log (|\mathcal{R}|/\beta )}{\epsilon }.$$

$${\mathcal{W}}_1=\{x\in \mathcal{X}\mid H(x)>H^{\prime }(x)\}$$

$${\Gamma }_{\zeta }(H,H^{\prime },D)=\{\begin{array}{ll}n& \text{if }{p}_1-p_2\le (2+\zeta )\alpha ;\\ n\cdot \max \{0,\hat{\tau }-(p_2+(1+\zeta /2)\alpha )\}& \text{otherwise.}\end{array}$$

$$S(H_j,D)=\underset{H_k\in \mathcal{H}}{\min }{\Gamma }_{\zeta }(H_j,H_k,D).$$

$$\mathrm{Pr}[\hat{H}=H_j]\propto \exp \left(\frac{S(H_j,D)}{2\epsilon }\right)$$

$$n\ge \frac{8\ln (4m/\beta )}{{\zeta }^2{\alpha }^2}+\frac{8\ln (2m/\beta )}{\zeta \alpha \epsilon }.$$

$$S(\hat{H},D)\ge S(H^{\ast },D)-\frac{2\ln (2m/\beta )}{\epsilon }>\zeta \alpha n/4-\frac{2\ln (2m/\beta )}{\epsilon }.$$

$$n\ge O\left(\frac{\log (m/\beta )+\log \log (1/\alpha )}{{\alpha }^2}+\frac{\log m+{\log }^2(1/\alpha )\cdot (\log (1/\beta )+\log \log (1/\alpha ))}{\alpha \epsilon }\right).$$

$$n\ge O\left(\frac{\log (m/\beta )}{{\alpha }^2}+\frac{m^2\log (m/\beta )}{\alpha \epsilon }\right).$$

$$d_{\mathrm{TV}}(\hat{H},P)\le 9\mathrm{OPT}+16\underset{H,H^{\prime }\in \mathcal{H}}{\max }\left|P({\mathcal{W}}_{H,H^{\prime }})-c_{H,H^{\prime }}\right|.$$

$$n\ge O\left(\frac{\log (m/\beta )+\log \log (1/\alpha )}{{\alpha }^2}+\frac{\log (m/\beta )+\log \log (1/\alpha )}{\alpha \epsilon }\right).$$

$$n\ge O\left(\frac{\log (1/\beta )+\log \log (1/\alpha )}{{\alpha }^2}+\frac{{\log }^2(1/\alpha )\cdot \left(\log (1/\beta )+\log \log (1/\alpha )\right)}{\alpha \epsilon }\right).$$

$${\mathrm{Pr}}_{D\leftarrow P^n}\left[\underset{f\in \mathcal{F}}{\sup }|f(D)-f(P)|\le \alpha \right]\ge 1-\beta$$

$$S(H,D)=\inf_{H^{\prime}\in\mathcal{H}}\max\left\{\begin{array}[]{c}|\{x\in D:H(x)>H^{\prime}(x)\}|-n\cdot(\mathrm{Pr}_{X\leftarrow H^{\prime}}\!\left[H(X)>H^{\prime}(X)\right]+3\alpha),\\ n\cdot\mathbb{I}[d_{\mathrm{TV}}(H,H^{\prime})\leq 6\alpha]\end{array}\right\},$$

$${\mathrm{Pr}}_{D\leftarrow P^n}\left[\forall H,H^{\prime }\in \mathcal{H}\left||\{x\in D:H(x)>H^{\prime }(x)\}|-n\cdot {\mathrm{Pr}}_{X\leftarrow P}\left[H(X)>H^{\prime }(X)\right]\right|\le \alpha n\right]\ge 1-\beta .$$

$$|\{x\in D:H(x)>H^{\prime }(x)\}|-n\cdot ({\mathrm{Pr}}_{X\leftarrow H^{\prime }}\left[H^{\ast }(X)>H^{\prime }(X)\right]+3\alpha )>\alpha n.$$

$$|\{x\in D:H(x)>H^{\prime }(x)\}|$$

$$\ge n\cdot ({\mathrm{Pr}}_{X\leftarrow H^{\ast }}\left[H(X)>H^{\prime }(X)\right]-d_{\mathrm{TV}}(P,H^{\ast })-\alpha )$$

$$\ge n\cdot (d_{\mathrm{TV}}(H^{\ast },H^{\prime })+{\mathrm{Pr}}_{X\leftarrow H^{\prime }}\left[H(X)>H^{\prime }(X)\right]-2\alpha )$$

$$>n\cdot (6\alpha +{\mathrm{Pr}}_{X\leftarrow H^{\prime }}\left[H(X)>H^{\prime }(X)\right]-2\alpha ),$$

$$|\{x\in D:H(x)>H^{\ast }(x)\}|-n\cdot ({\mathrm{Pr}}_{X\leftarrow H^{\ast }}\left[H(X)>H^{\ast }(X)\right]+3\alpha )\le 0.$$

$$|\{x\in D:H(x)>H^{\ast }(x)\}|$$

$$\le n\cdot ({\mathrm{Pr}}_{X\leftarrow H^{\ast }}\left[H(X)>H^{\ast }(X)\right]+d_{\mathrm{TV}}(P,H^{\ast })+\alpha )$$

$$\le n\cdot ({\mathrm{Pr}}_{X\leftarrow H^{\ast }}\left[H(X)>H^{\ast }(X)\right]+2\alpha ),$$

$$K(D,5\alpha ):=\left|\left\{H\in \mathcal{H}:S(H,D)\ge \underset{H^{\prime }\in \mathcal{H}}{\sup }S(H^{\prime },D)-5\alpha n\right\}\right|.$$

$$K(D,5\alpha )\le k⟹\mathrm{Pr}\left[S(M(D),D)\ge \underset{H^{\prime }\in \mathcal{H}}{\sup }S(H^{\prime },D)-\alpha n\right]\ge 1-\beta$$

$$\mathrm{Pr}\left[B=b\right]\propto \exp \left(\frac{\epsilon }{4}\sup \left\{S(H,D):H\in \mathcal{H},G(H)=b\right\}\right).$$

Full text

We apply our hypothesis selection algorithm to give learning algorithms for a number of natural distribution classes, including Gaussians, product distributions, sums of independent random variables, piecewise polynomials, and mixture classes. Our hypothesis selection procedure allows us to generically convert a cover for a class to a learning algorithm, complementing known learning lower bounds which are in terms of the size of the packing number of the class. As the covering and packing numbers are often closely related, for constant $\alpha$, our algorithms achieve the optimal sample complexity for many classes of interest. Finally, we describe an application to private distribution-free PAC learning.

1 Introduction

We consider the problem of hypothesis selection: given samples from an unknown probability distribution, select a distribution from some fixed set of candidates which is “close” to the unknown distribution in some appropriate distance measure. Such situations can arise naturally in a number of settings. For instance, we may have a number of different methods which work under various circumstances, which are not known in advance. One option is to run all the methods to generate a set of hypotheses, and pick the best from this set afterwards. Relatedly, an algorithm may branch its behavior based on a number of “guesses,” which will similarly result in a set of candidates, corresponding to the output at the end of each branch. Finally, if we know that the underlying distribution belongs to some (parametric) class, it is possible to essentially enumerate the class (also known as a cover) to create a collection of hypotheses. Observe that this last example is quite general, and this approach can give generic learning algorithms for many settings of interest.

This problem of hypothesis selection has been extensively studied (see, e.g., [Yat85, DL96, DL97, DL01]), resulting in algorithms with a sample complexity which is logarithmic in the number of hypotheses. Such a mild dependence is critical, as it facilitates sample-efficient algorithms even when the number of candidates may be large. These initial works have triggered a great deal of study into hypothesis selection with additional considerations, including computational efficiency, understanding the optimal approximation factor, adversarial robustness, and weakening access to the hypotheses (e.g., [MS08, DDS12b, DK14, SOAJ14, AJOS14, DKK*+*16, AFJ*+*18, BKM19]).

However, in modern settings of data analysis, data may contain sensitive information about individuals. Some examples of such data include medical records, GPS location data, or private message transcripts. As such, we would like to perform statistical inference in these settings without revealing significant information about any particular individual’s data. To this end, there have been many proposed notions of data privacy, but perhaps the gold standard is that of differential privacy [DMNS06]. Informally, differential privacy requires that, if a single datapoint in the dataset is changed, then the distribution over outputs produced by the algorithm should be similar (see Definition 2.4). Differential privacy has seen widespread adoption, including deployment by Apple [Dif17], Google [EPK14], and the US Census Bureau [DLS*+*17].

This naturally raises the question of whether one can perform hypothesis selection under the constraint of differential privacy, while maintaining a logarithmic dependence on the size of the cover. Such a tool would allow us to generically obtain private learning results for a wide variety of settings.

1.1 Results

Our main results answer this in the affirmative: we provide differentially private algorithms for selecting a good hypothesis from a set of distributions. The output distribution is competitive with the best distribution, and the sample complexity is bounded by the logarithm of the size of the set. The following is a basic version of our main result.

Theorem 1.1**.**

Let $\mathcal{H}=\{H_{1},\dots,H_{m}\}$ be a set of probability distributions. Let $D=\{X_{1},\dots,X_{n}\}$ be a set of samples drawn independently from an unknown probability distribution $P$. There exists an $\varepsilon$-differentially private algorithm (with respect to the dataset $D$) which has following guarantees. Suppose there exists a distribution $H^{*}\in\mathcal{H}$ such that $d_{\mathrm{TV}}(P,H^{*})\leq\alpha$. If $n=\Omega\left(\frac{\log m}{\alpha^{2}}+\frac{\log m}{\alpha\varepsilon}\right)$, then the algorithm will output a distribution $\hat{H}\in\mathcal{H}$ such that $d_{\mathrm{TV}}(P,\hat{H})\leq(3+\zeta)\alpha$ with probability at least $9/10$, for any constant $\zeta>0$. The running time of the algorithm is $O(nm^{2})$.

The sample complexity of this problem without privacy constraints is $O\left(\frac{\log m}{\alpha^{2}}\right)$, and thus the additional cost for $\varepsilon$-differential privacy is an additive $O\left(\frac{\log m}{\alpha\varepsilon}\right)$. We consider this cost to be minimal; in particular, the dependence on $m$ is unchanged. Note that the running time of our algorithm is $O(nm^{2})$ – we conjecture it may be possible to reduce this to $\tilde{O}(nm)$ as has been done in the non-private setting [DK14, SOAJ14, AJOS14, AFJ*+*18], though we have not attempted to perform this optimization. Regardless, our main focus is on the sample complexity rather than the running time, since any method for generic hypothesis selection requires $\Omega(m)$ time, thus precluding efficient algorithms when $m$ is large. Note that the approximation factor of $(3+\zeta)\alpha$ is effectively tight. That is, even in the infinite sample limit and without the constraint of privacy, information theoretically, one can not achieve a better approximation than $3\alpha$ [DL01, MS08].111Note that this can be brought down to $2\alpha$ if one instead outputs a mixture of $H_{i}\in\mathcal{H}$ [BKM19]. Theorem 1.1 requires prior knowledge of the value of $\alpha$, though we can use this to obtain an algorithm with similar guarantees which does not (Theorem 3.5).

It is possible to improve the guarantees of this algorithm in two ways (Theorem 4.1). First, if the distributions are nicely structured, the former term in the sample complexity can be reduced from $O(\log m/\alpha^{2})$ to $O(d/\alpha^{2})$, where $d$ is a VC-dimension-based measure of the complexity of the collection of distributions. Second, if there are few hypotheses which are close to the true distribution, then we can pay only logarithmically in this number, as opposed to the total number of hypotheses. These modifications allow us to handle instances where $m$ may be very large (or even infinite), albeit at the cost of weakening to approximate differential privacy to perform the second refinement. A technical discussion of our methods is in Section 1.2, our basic approach is covered in Section 3, and the version with all the bells and whistles appears in Section 4.

From Theorem 1.1, we immediately obtain Corollary 1.2 which applies when $\mathcal{H}$ itself may not be finite, but admits a finite cover with respect to total variation distance.

Corollary 1.2**.**

Suppose there exists an $\alpha$-cover $\mathcal{C}_{\alpha}$ of a set of distributions $\mathcal{H}$, and that we are given a set of samples $X_{1},\dots,X_{n}\sim P$, where $d_{\mathrm{TV}}(P,\mathcal{H})\leq\alpha$. For any constant $\zeta>0$, there exists an $\varepsilon$-differentially private algorithm (with respect to the input $\{X_{1},\dots,X_{n}\}$) which outputs a distribution $H^{*}\in\mathcal{C}_{\alpha}$ such that $d_{\mathrm{TV}}(P,H^{*})\leq(6+2\zeta)\alpha$ with probability $\geq 9/10$, as long as

[TABLE]

Informally, this says that if a hypothesis class has an $\alpha$-cover $\mathcal{C}_{\alpha}$, then there is a private learning algorithm for the class which requires $O(\log|\mathcal{C}_{\alpha}|)$ samples. Note that our algorithm works even if the unknown distribution is only close to the hypothesis class. This is useful when we may have model misspecification, or when we require adversarial robustness. (We also give an extension of this algorithm which gives guarantees in the semi-agnostic learning model; see Section 3.4 for details.) The requirements for this theorem to apply are minimal, and thus it generically provides learning algorithms for a wide variety of hypothesis classes. That said, in non-private settings, the sample complexity given by this method is rather lossy: as an extreme example, there is no finite-size cover of univariate Gaussian distributions with unbounded parameters, so this approach does not give a finite-sample algorithm. That said, it is well-known that $O(1/\alpha^{2})$ samples suffice to estimate a Gaussian in total variation distance. In the private setting, our theorem incurs a cost which is somewhat necessary: in particular, it is folklore that any pure $\varepsilon$-differentially private learning algorithm must pay a cost which is logarithmic in the packing number of the class (for completeness, see Lemma 5.1). Due to the relationship between packing and covering numbers (Lemma 5.2), this implies that up to a constant factor relaxation in the learning accuracy, our results are tight (Theorem 5.3). Further discussion appears in Sections 5.

Given Corollary 1.2, in Section 6, we derive new learning results for a number of classes. Our main applications are for $d$-dimensional Gaussian and product distributions. Informally, we obtain $\tilde{O}(d)$ sample algorithms for learning a product distribution and a Gaussian with known covariance (Corollaries 6.3 and 6.10), and an $\tilde{O}(d^{2})$ algorithm for learning a Gaussian with unknown covariance (Corollary 6.11). These improve on recent results by Kamath, Li, Singhal, and Ullman [KLSU19] in two different ways. First, as mentioned before, our results are semi-agnostic, so we can handle when the distribution is only close to a product or Gaussian distribution. Second, our results hold for pure $(\varepsilon,0)$-differential privacy, which is a stronger notion than $\varepsilon^{2}$-zCDP as considered in [KLSU19]. In this weaker model, they also obtained $\tilde{O}(d)$ and $\tilde{O}(d^{2})$ sample algorithms, but the natural modifications to achieve $\varepsilon$-DP incur extra $\operatorname*{poly}(d)$ factors.222Roughly, this is due to the fact that the Laplace and Gaussian mechanism are based on $\ell_{1}$ and $\ell_{2}$ sensitivity, respectively, and that there is a $\sqrt{d}$-factor relationship between these two norms, in the worst case. [KLSU19] also showed $\tilde{\Omega}(d)$ lower bounds for Gaussian and product distribution estimation in the even weaker model of $(\varepsilon,\delta)$-differential privacy. Thus, our results show that the dimension dependence for these problems is unchanged for essentially any notion of differential privacy. In particular, our results show a previously-unknown separation between mean estimation of product distributions and non-product distributions under pure $(\varepsilon,0)$-differential privacy; see Remark 6.4.

We also apply Theorem 4.1 to obtain algorithms for learning Gaussians under $(\varepsilon,\delta)$-differential privacy, with no bounds on the mean and variance parameters. More specifically, we provide algorithms for learning multivariate Gaussians with unknown mean and known covariance (Corollary 6.13), and univariate Gaussians with both unknown mean and variance (Corollary 6.15). For the former problem, we manage to avoid dependences which arise due to the application of advanced composition (similar to Remark 6.4).

To demonstrate the flexibility of our approach, we also give private learning algorithms for sums of independent random variables (Corollaries 6.20 and 6.22) and piecewise polynomials (Corollary 6.29). To the best of our knowledge, the former class of distributions has not been considered in the private setting, and we rely on covering theorems from the non-private literature. Private learning algorithms for the latter class, piecewise polynomials, have been studied by Diakonikolas, Hardt, and Schmidt [DHS15]. They provide sample and time efficient algorithms for histogram distributions (i.e., piecewise constant distributions), and claim similar results for general piecewise polynomials. Their method depends heavily on rather sophisticated algorithms for the non-private version of this problem [ADLS17]. In constrast, we can obtain comparable sample complexity bounds from just the existence of a cover and elementary VC dimension arguments, which we derive in a fairly self-contained manner.

We additionally give algorithms for learning mixtures of any coverable class (Corollary 6.32). In particular, this immediately implies algorithms for learning mixtures of Gaussians, product distributions, and all other classes mentioned above.

To conclude our applications, we discuss a connection to PAC learning (Corollary 6.34). It is known that the sample complexity of differentially private distribution-free PAC learning can be higher than that of non-private learning. However, this gap does not exist for distribution-specific learning, where the learning algorithm knows the distribution of (unlabeled) examples, as both sample complexities are characterized by VC dimension. Private hypothesis selection allows us to address an intermediate situation where the distribution of unlabeled examples is not known exactly, but is known to come (approximately) from a class of distributions. When this class has a small cover, we are able to recover sample complexity guarantees for private PAC learning which are comparable to the non-private case.

1.2 Techniques

Non-privately, most algorithms for hypothesis selection involve a tournament-style approach. We conduct a number of pairwise comparisons between distributions, which may either have a winner and a loser, or may be declared a draw. Intuitively, a distribution will be declared the winner of a comparison if it is much closer than the alternative to the unknown distribution, and a tie will be declared if the two distributions are comparably close. The algorithm will output any distribution which never loses a comparison. A single comparison between a pair of hypotheses requires $O(1/\alpha^{2})$ samples, and a Chernoff plus union bound argument over the $O(m^{2})$ possible comparisons increases the sample complexity to $O(\log m/\alpha^{2})$. In fact, we can use uniform convergence arguments to reduce this sample complexity to $O(d/\alpha^{2})$, where $d$ is the VC dimension of the $2\binom{m}{2}$ sets (the “Scheffé” sets) defined by the subsets of the domain where the PDF of one distribution dominates another. Crucially, we must reuse the same set of samples for all comparisons to avoid paying polynomially in the number of hypotheses.

A private algorithm for this problem requires additional care. Since a single comparison is based on the number of samples which fall into a particular subset of the domain, the sensitivity of the underlying statistic is low, and thus privacy may seem easily achievable at first glance. However, the challenge comes from the fact that the same samples are reused for all pairwise comparisons, thus greatly increasing the sensitivity: changing a single datapoint could flip the result of every comparison! In order to avoid this pitfall, we instead carefully construct a score function for each hypothesis, namely, the minimum number of points that must be changed to cause the distribution to lose any comparison. For this to be a useful score function, we must show that the best hypothesis will win all of its comparisons by a large margin. We can then use the Exponential Mechanism [MT07] to select a distribution with high score.

Further improvements can be made if we are guaranteed that the number of “good” hypotheses (i.e., those that have total variation distance from the true distribution bounded by $(3+\zeta)\alpha$) is at most some parameter $k$, and if we are willing to relax to approximate differential privacy. The parameter $k$ here is related to the doubling dimension of the hypothesis class with respect to total variation distance. If we randomly assign the hypotheses to $\Omega(k^{2})$ buckets, with high probability, no bucket will contain more than one good hypothesis. We can identify a bucket containing a good hypothesis using a similar method based on the exponential mechanism as described above. Moreover, since we are likely to only have one “good” hypothesis in the chosen bucket, this implies a significant gap between the best and second-best scores in that bucket. This allows us to use stability-based techniques [DL09, TS13], and in particular the GAP-MAX algorithm of Bun, Dwork, Rothblum, and Steinke [BDRS18], to identify an accurate distribution.

1.3 Related Work

Our main result builds on a long line of work on non-private hypothesis selection. One starting point for the particular style of approach we consider here is [Yat85], which was expanded on in [DL96, DL97, DL01]. Since then, there has been study into hypothesis selection under additional considerations, including computational efficiency, understanding the optimal approximation factor, adversarial robustness, and weakening access to the hypotheses [MS08, DDS12b, DK14, SOAJ14, AJOS14, DKK*+*16, AFJ*+*18, BKM19]. Our private algorithm examines the same type of problem, with the additional constraint of differential privacy.

Perhaps the most closely related work is that of Canonne, Kamath, McMillan, Smith, and Ullman [CKM*+*19], which focuses on the case of private simple hypothesis testing. This is a more restricted setting than we consider in this paper, as it focuses on the case where we are trying to decide between $m=2$ hypotheses, and we are guaranteed that the unknown distribution is one of these two hypotheses. However, in this setting, they are able to get an instance-by-instance characterization of the sample complexity, depending on both the total variation and Hellinger distance between the two distributions.

There has recently been a great deal of interest in differentially private distribution learning. In the central model, most relevant are [DHS15], which gives algorithms for learning structured univariate distributions, and [KV18, KLSU19], which focus on learning Gaussians and binary product distributions. [CWZ19] also studies private statistical parameter estimation. Privately learning mixtures of Gaussians was considered in [NRS07, KSSU19]. The latter paper (which is concurrent with the present work) gives a computationally efficient algorithm for the problem, but with a worse sample complexity, and incomparable accuracy guarantees (they require a separation condition, and perform clustering and parameter estimation, while we do proper learning). [BNSV15] give an algorithm for learning distributions in Kolmogorov distance. Upper and lower bounds for learning the mean of a product distribution over the hypercube in $\ell_{\infty}$-distance include [BDMN05, BUV14, DMNS06, SU17]. [AKSZ18] focuses on estimating properties of a distribution, rather than the distribution itself. [Smi11] gives an algorithm which allows one to estimate asymptotically normal statistics with optimal convergence rates, but no finite sample complexity guarantees. There has also been a great deal of work on distribution learning in the local model of differential privacy [DJW13, WHW*+*16, KBR16, ASZ19, DR18, JKMW19, YB18, GRS19]. For further coverage of differentially private statistics, see [KU20].

Non-privately, there has been a significant amount of work on learning specific classes of distributions. The PAC-style formulation of the problem we consider originated in [KMR*+*94]. While learning Gaussians and product distributions can be considered folklore at this point, some of the other classes we learn have enjoyed more recent study. For instance, learning sums of independent random variables was recently considered in [DDS12b] toward the problem of learning Poisson Binomial Distributions (PBDs). Since then, there has been additional work on learning PBDs and various generalizations [DKT15, DDKT16, DKS16b, DKS16c, DKS16a, DLS18].

Piecewise polynomials are a highly-expressive class of distributions, and they can be used to approximate a number of other univariate distribution classes, including distributions which are multi-modal, concave, convex, log-concave, monotone hazard rate, Gaussian, Poisson, Binomial, and more. Algorithms for learning such classes are considered in a number of papers, including [DDS12a, CDSS14a, CDSS14b, ADK15, ADLS17].

There has also been a great deal of work on learning mixtures of distribution classes, particularly mixtures of Gaussians. There are many ways the objective of such a problem can be defined, including clustering [Das99, DS00, AK01, VW02, AM05, CR08b, CR08a, KK10, AS12, RV17, HL18, DKS18, KSS18], parameter estimation [KMV10, MV10, BS10, HK13, ABG*+*14, BCMV14, HP15, GHK15, XHM16, DTZ17, ABDH*+*18], proper learning [FOS06, FOS08, DK14, SOAJ14, DKK*+*16, LS17], and improper learning [CDSS14a]. Our work falls into the line on proper learning: the algorithm is given a set of samples from a mixture of Gaussians, and must output a mixture of Gaussians which is close in total variation distance.

1.3.1 Subsequent Work

Since the initial appearance of this work, there have been several relevant results in the surrounding area. Most pertinent is the work of Aden-Ali, Ashtiani, and Kamath [AAAK21], which improves our main hypothesis selection result by improving the constant approximation factor, making the algorithm agnostic, and having a simpler analysis. The authors also argue that, given a small cover for each distribution in the space, there exists a cover which is locally-small everywhere as required by Theorem 4.1, thus allowing them to learn unbounded Gaussians with arbitrary covariances.

Our results and techniques have seen use for other problems as well. Kamath, Singhal, and Ullman [KSU20] use a similar approach based on pairwise comparisons in order to perform mean estimation, rather than our work which focuses on density estimation. Liu et al. [LSY*+*20] use our main algorithm to estimate discrete distributions in a type of federated learning setting.

Hypothesis selection has since been studied in other privacy models. Informally speaking, our work shows that the sample complexity of hypothesis selection under central differential privacy is $O(\log m)$. Gopi et al. [GKK*+*20] study the problem under the stronger notion of local differential privacy [War65, EGS03, KLN*+*11], showing that the sample complexity is $\tilde{\Theta}(m)$, an exponential increase in cost. Under various notions of pan-privacy [DNP*+*10] and the shuffled model [CSU*+*19, EFM*+*19], which are intermediate to local and central differential privacy, the sample complexity of $m$-wise simple hypothesis testing (the easier version of hypothesis selection, where the unknown distribution is equal to one of the given distributions) was shown to be $\tilde{\Theta}(\sqrt{m})$ [CU20].

1.4 Organization

We begin in Section 2 with preliminaries. In Section 3, we give a basic algorithm for private hypothesis selection, via the exponential mechanism. In Section 4, we extend this approach in two ways: by using VC dimension arguments to reduce the sample complexity for sets of hypotheses with additional structure, and combining this with a GAP-MAX algorithm to achieve non-trivial guarantees for infinite hypothesis classes. Section 5 shows that our approach leads to algorithms which essentially match lower bounds for most distribution classes (in the constant $\alpha$ regime). We consider applications in Section 6: through a combination of arguments about covers and VC dimension, we derive algorithms for learning a number of classes of distributions, as well as describe an application to private PAC learning. Finally, we conclude in Section 7 with open questions.

2 Preliminaries

We start with some preliminaries and definitions.

Definition 2.1**.**

The total variation distance or statistical distance between $P$ and $Q$ is defined as

[TABLE]

Moreover, if $\mathcal{H}$ is a set of distributions over a common domain, we define $d_{\mathrm{TV}}(P,\mathcal{H})=\inf_{H\in\mathcal{H}}d_{\mathrm{TV}}(P,H)$.

Throughout this paper, we consider packings and coverings of sets of distributions with respect to total variation distance.

Definition 2.2**.**

A $\gamma$-cover of a set of distributions $\mathcal{H}$ is a set of distributions $\mathcal{C}_{\gamma}$, such that for every $H\in\mathcal{H}$, there exists some $P\in\mathcal{C}_{\gamma}$ such that $d_{\mathrm{TV}}(P,H)\leq\gamma$.

A $\gamma$-packing of a set of distributions $\mathcal{H}$ is a set of distributions $\mathcal{P}_{\gamma}\subseteq\mathcal{H}$, such that for every pair of distributions $P,Q\in\mathcal{P}_{\gamma}$, we have that $d_{\mathrm{TV}}(P,Q)>\gamma$.

In this paper, we present semi-agnostic learning algorithms.

Definition 2.3**.**

An algorithm is said to be an $\alpha$-semi-agnostic learner for a class $\mathcal{H}$ if it has the following guarantees. Suppose we are given $X_{1},\dots,X_{n}\sim P$, where $d_{\mathrm{TV}}(P,\mathcal{H})\leq\mathrm{OPT}$. The algorithm must output some distribution $\hat{H}$ such that $d_{\mathrm{TV}}(P,H)\leq c\cdot\mathrm{OPT}+O(\alpha)$, for some constant $c\geq 1$. If $c=1$, then the algorithm is said to be agnostic.

Now we define differential privacy. We say that $D$ and $D^{\prime}$ are neighboring datasets, denoted $D\sim D^{\prime}$, if $D$ and $D^{\prime}$ differ by at most one observation. Informally, differential privacy requires that the algorithm has close output distributions when run on any pair of neighboring datasets. More formally:

Definition 2.4** ([DMNS06]).**

A randomized algorithm $T:X^{*}\rightarrow\mathcal{R}$ is $(\varepsilon,\delta)$-differentially private if for all $n\geq 1$, for all neighboring datasets $D,D^{\prime}\in X^{n}$, and for all events $S\subseteq\mathcal{R}$,

[TABLE]

If $\delta=0$, we say that $T$ is $\varepsilon$-differentially private.

We will also use the related notion of concentrated differential privacy:

Definition 2.5** ([DR16, BS16]).**

A randomized algorithm $T:X^{*}\rightarrow\mathcal{R}$ satisfies $\rho$-zero-concentrated differential privacy if for all $n\geq 1$, for all neighboring datasets $D,D^{\prime}\in X^{n}$, and for all $\alpha\in(1,\infty)$,

[TABLE]

where $R_{\alpha}(M(D)||M(D^{\prime}))$ is the $\alpha$-Rényi divergence between $M(D)$ and $M(D^{\prime})$.333Given two probability distributions $P,Q$ over $\Omega$, $R_{\alpha}(P||Q)=\frac{1}{\alpha-1}\log\left(\sum_{x\in\Omega}P(x)^{\alpha}Q(x)^{1-\alpha}\right)$.

The exponential mechanism [MT07] is a powerful $\varepsilon$-differentially private mechanism for selecting an approximately best outcome from a set of alternatives, where the quality of an outcome is measured by a score function relating each alternative to the underlying dataset. Letting $\mathcal{R}$ be the set of possible outcomes, a score function $q:X^{*}\times\mathcal{R}\rightarrow\mathbb{R}$ maps each pair consisting of a dataset and an outcome to a real-valued score. The exponential mechanism $\mathcal{M}_{E}$ instantiated with a dataset $D$, a score function $q$, and a privacy parameter $\varepsilon$ selects an outcome $r$ in $\mathcal{R}$ with probability proportional to $\exp\left(\varepsilon q(D,r)/(2\Delta(q))\right)$, where $\Delta(q)$ is the sensitivity of the score function defined as

[TABLE]

Theorem 2.6** ([MT07]).**

For any input dataset $D$, score function $q$ and privacy parameter $\varepsilon>0$, the exponential mechanism $\mathcal{M}_{E}(D,q,\varepsilon)$ is $\varepsilon$-differentially private, and with probability at least $1-\beta$, selects an outcome $r\in\mathcal{R}$ such that

[TABLE]

3 A First Method for Private Hypothesis Selection

In this section, we present our first algorithm for private hypothesis selection and obtain the following result.

See 1.1

Note that the sample complexity bound above scales logarithmically with the size of the hypothesis class. In Section 4, we will provide a stronger result (which subsumes the present one as a special case) that can handle certain infinite hypothesis classes. For sake of exposition, we begin in this section with the basic algorithm.

3.1 Pairwise Comparisons

We first present a subroutine which compares two hypothesis distributions. This subroutine is due to Daskalakis, Diakonikolas, and Servedio [DDS12b], and is essentially a modification of previous methods (e.g., [DL01]) to allow for draws. Let $H$ and $H^{\prime}$ be two distributions over domain $\mathcal{X}$ and consider the following set, which is called the Scheffé set:

[TABLE]

Define $p_{1}=H(\mathcal{W}_{1})$, $p_{2}=H^{\prime}(\mathcal{W}_{1})$, and $\tau=P(\mathcal{W}_{1})$ to be the probability masses that $H$, $H^{\prime}$, and $P$ place on $\mathcal{W}_{1}$, respectively. It follows that $p_{1}>p_{2}$ and $p_{1}-p_{2}=d_{\mathrm{TV}}(H,H^{\prime})$.444For simplicity of our exposition, we will assume that we can evaluate the two quantities $p_{1}$ and $p_{2}$ exactly. In general, we can estimate these quantities to arbitrary accuracy, as long as, for each hypothesis $H$, we can evaluate the density of each point under $H$ and also draw samples from $H$.

Now consider the following function of this ordered pair of hypotheses:

[TABLE]

When the two hypotheses are sufficiently far apart (i.e., $d_{\mathrm{TV}}(H,H^{\prime})>(2+\zeta)\alpha$), $\Gamma_{\zeta}(H,H^{\prime},D)$ is essentially the number of points one needs to change in $D$ to make $H^{\prime}$ the winner.

Lemma 3.1**.**

Let $P,H,H^{\prime}$ be distributions as above. With probability at least $1-2\exp(-n\zeta^{2}\alpha^{2}/8)$ over the random draws of $D$ from $P^{n}$, $\hat{\tau}$ satisfies $|\hat{\tau}-\tau|<\zeta\alpha/4$, and if $d_{\mathrm{TV}}(P,H)\leq\alpha$, then $\Gamma_{\zeta}(H,H^{\prime},D)>\zeta\alpha n/4$.

Proof.

By applying Hoeffding’s inequality, we know that with probability at least $1-2\exp(-n\zeta^{2}\alpha^{2}/8)$, $|\tau-\hat{\tau}|<\zeta\alpha/4$. We condition on this event for the remainder of the proof. Consider the following two cases. In the first case, suppose that $p_{1}-p_{2}\leq(2+\zeta)\alpha$. Then we know that $\Gamma_{\zeta}(H,H^{\prime},D)=n>\alpha n$. In the second case, suppose that $p_{1}-p_{2}>(2+\zeta)\alpha$. Since $d_{\mathrm{TV}}(P,H)\leq\alpha$, we know that $|p_{1}-\tau|\leq\alpha$, and so $|p_{1}-\hat{\tau}|<(1+\zeta/4)\alpha$. Since $p_{1}>p_{2}+(2+\zeta)\alpha$, we also have $\hat{\tau}>p_{2}+(1+3\zeta/4)\alpha$. It follows that $\Gamma_{\zeta}(H,H^{\prime},D)=n(\hat{\tau}-(p_{2}+(1+\zeta/2)\alpha))>\zeta\alpha n/4$. ∎

3.2 Naïve Approach via Laplace Mechanism

We first sketch a naïve approach for private hypothesis selection, based on the primitive in Algorithm 1. This is a privatization of a similar approach which appeared in [DK14], though the idea behind the approach is older, e.g., [DL01] – the [DK14] approach differs slightly since it employs a comparison procedure which allows ties, as we do. Later, Lemma 3.6 describes the approach and privatization of [DL01] in more detail, which are morally equivalent to what we discuss here.

A non-private algorithm for selection from $m$ hypotheses would run Algorithm 1 on each pair of hypotheses, either outputting a winner between the two distributions, or declaring a tie in the case when the total variation distance between the two distributions is small. The algorithm would output any distribution which never loses a comparison. Correctness of this algorithm relies on the empirical masses in all $O(m^{2})$ Scheffé sets being estimated up to an additive $O(\alpha)$, which, by Hoeffding’s inequality, happens with constant probability when $n\geq O\left(\frac{\log m}{\alpha^{2}}\right)$. Crucially, we reuse the same set of samples for all comparisons. With this in hand, it is not hard to show that a distribution $H$ which is $\alpha$-close to $P$ will never lose a comparison, and any distribution $H^{\prime}$ which is $c\alpha$-far from $P$ (for an appropriately chosen constant $c>1$) will lose its comparison with $H$, thus ensuring that the winning distribution will be $c\alpha$-close to $P$.

Now, we consider how to privatize this algorithm. Each of the $O(m^{2})$ comparisons is based on the quantity $\hat{\tau}=\frac{1}{n}\left|\{x\in D\mid x\in\mathcal{W}\}\right|$, where $\mathcal{W}$ is the Scheffé set between the two distributions $H$ and $H^{\prime}$. To make a single comparison $\varepsilon$-differentially private, we would have to add Laplace noise of order $O\left(\frac{1}{\varepsilon n}\right)$ to this quantity. However, since we reuse the same set of samples for all comparisons, in order to make the result of all $O(m^{2})$ comparisons $\varepsilon$-differentially private, the basic composition property of differential privacy would prescribe adding Laplace noise of order $O\left(\frac{m^{2}}{\varepsilon n}\right)$ to the quantity used in each comparison. To bound the noise error of all comparisons simultaneously by $O(\alpha)$, we thus require $n\geq O\left(\frac{m^{2}\log m}{\alpha\varepsilon}\right)$, and the rest of the analysis is then identical to before.

A formalization of this argument allows us to arrive at the following theorem. The accuracy bound is of the appropriate form, but the cost of privacy is an exponential increase in the sample complexity.

Theorem 3.2**.**

Let $\mathcal{H}=\{H_{1},\dots,H_{m}\}$ be a set of probability distributions. Let $D=\{X_{1},\dots,X_{n}\}$ be a set of samples drawn independently from an unknown probability distribution $P$. There exists an $\varepsilon$-differentially private algorithm (with respect to the dataset $D$) which has following guarantees. Suppose there exists a distribution $H^{*}\in\mathcal{H}$ such that $d_{\mathrm{TV}}(P,H^{*})\leq\alpha$. If $n=\Omega\left(\frac{\log m}{\alpha^{2}}+\frac{m^{2}\log m}{\alpha\varepsilon}\right)$, then the algorithm will output a distribution $\hat{H}\in\mathcal{H}$ such that $d_{\mathrm{TV}}(P,\hat{H})\leq O(\alpha)$ with probability at least $9/10$. The running time of the algorithm is $O(nm^{2})$.

3.3 Selection via Exponential Mechanism

In light of the definition of the pairwise comparison defined above, we consider the following score function $S\colon\mathcal{H}\times\mathcal{X}^{n}$, such that for any $H_{j}\in\mathcal{H}$ and dataset $D$,

[TABLE]

Roughly speaking, $S(H_{j},D)$ is the minimum number of points required to change in $D$ in order for $H_{j}$ to lose at least one pairwise contest against a different hypothesis. When the hypothesis $H_{j}$ is very close to every other distribution, such that all pairwise contests return “Draw,” then the score will be $n$.

Lemma 3.3** (Privacy).**

For any $\varepsilon>0$ and collection of hypotheses $\mathcal{H}$, the algorithm $\text{PHS}(\mathcal{H},\cdot,\varepsilon)$ satisfies $\varepsilon$-differential privacy.

Proof.

First, observe that for any pairs of hypotheses $H_{j},H_{k}$, $\Gamma_{\zeta}(H_{j},H_{k},\cdot)$ has sensitivity 1. As a result, the score function $S$ is also 1-sensitive. Then the result directly follows from the privacy guarantee of the exponential mechanism (Theorem 2.6). ∎

Lemma 3.4** (Utility).**

Fix any $\alpha,\beta\in(0,1)$, and $\zeta>0$. Suppose that there exists $H^{*}\in\mathcal{H}$ such that $d_{\mathrm{TV}}(P,H^{*})\leq\alpha$. Then with probability $1-\beta$ over the sample $D$ and the algorithm PHS, we have that $\text{PHS}(\mathcal{H},D)$ outputs an hypothesis $\hat{H}$ such that $d_{\mathrm{TV}}(P,\hat{H})\leq(3+\zeta)\alpha$, as long as the sample size satisfies

[TABLE]

Proof.

First, consider the $m$ pairwise contests between $H^{*}$ and every candidate in $\mathcal{H}$. Let $\mathcal{W}_{j}=\{x\in\mathcal{X}\mid H^{*}(x)>H_{j}(x)\}$ be the collection of Scheffé sets. For any event $W\subseteq\mathcal{X}$, let $\hat{P}(W)$ denote the empirical probability of event $W$ on the dataset $D$. By Lemma 3.1 and an application of the union bound, we know that with probability at least $1-2m\exp(-n\zeta^{2}\alpha^{2}/8)$ over the draws of $D$, $|P(\mathcal{W}_{j})-\hat{P}(\mathcal{W}_{j})|\leq\zeta\alpha/4$ and $\Gamma_{\zeta}(H^{*},H_{j},D)>\zeta\alpha n/4$ for all $H_{j}\in\mathcal{H}$. In particular, the latter event implies that $S(H^{*},D)>\zeta\alpha n/4$.

Next, by the utility guarantee of the exponential mechanism (Theorem 2.6), we know that with probability at least $1-\beta/2$, the output hypothesis satisfies

[TABLE]

Then as long as $n\geq\frac{8\ln(4m/\beta)}{\zeta^{2}\alpha^{2}}+\frac{8\ln(2m/\beta)}{\zeta\alpha\varepsilon}$, we know that with probability at least $1-\beta$, $S(\hat{H},D)>0$. Let us condition on this event, which implies that $\Gamma_{\zeta}(\hat{H},H^{*},D)>0$. We will now show that $d_{\mathrm{TV}}(\hat{H},H^{*})\leq(2+\zeta)\alpha$, which directly implies that $d_{\mathrm{TV}}(\hat{H},P)\leq(3+\zeta)\alpha$ by the triangle inequality. Suppose to the contrary that $d_{\mathrm{TV}}(\hat{H},H^{*})>(2+\zeta)\alpha$. Then by the definition of $\Gamma_{\zeta}$, $\hat{P}(\hat{\mathcal{W}})>H^{*}(\hat{\mathcal{W}})+(1+\zeta/2)\alpha$, where $\hat{\mathcal{W}}=\{x\in\mathcal{X}\mid\hat{H}(x)>H^{*}(x)\}$. Since $|P(\hat{\mathcal{W}})-\hat{P}(\hat{\mathcal{W}})|\leq\zeta\alpha/4$, we have $P(\hat{\mathcal{W}})>H^{*}(\hat{\mathcal{W}})+(1+\zeta/4)\alpha$, which is a contradiction to the assumption that $d_{\mathrm{TV}}(P,H^{*})\leq\alpha$. ∎

3.4 Obtaining a Semi-Agnostic Algorithm

Theorem 1.1 shows that given a hypothesis class $\mathcal{H}$ and samples from an unknown distribution $P$, we can privately find a distribution $\hat{H}\in\mathcal{H}$ with $d_{\mathrm{TV}}(P,\hat{H})\leq(3+\zeta)\alpha$ provided that we know $d_{\mathrm{TV}}(P,\mathcal{H})\leq\alpha$. But what if we are not promised that $P$ is itself close to $\mathcal{H}$? We would like to design a private hypothesis selection algorithm for the more general semi-agnostic setting, where for any value of $\operatorname{OPT}:=d_{\mathrm{TV}}(P,\mathcal{H})$, we are able to privately identify a distribution $\hat{H}\in\mathcal{H}$ with $d_{\mathrm{TV}}(P,\hat{H})\leq c\cdot\operatorname{OPT}+\alpha$ for some universal constant $c$. Our goal will be to do this with sample complexity which is still logarithmic in $|\mathcal{H}|$.

Our strategy for handling this more general setting is by a reduction to that of Theorem 1.1. We run that algorithm $T=O(\log(1/\alpha))$ times, doubling the choice of $\alpha$ in each run and producing a sequence of candidate hypotheses $H_{1},\dots,H_{T}$. By the guarantees of Theorem 1.1, there is some candidate $H_{t}$ with $d_{\mathrm{TV}}(P,H_{t})\leq 2(3+\zeta)\operatorname{OPT}$. The remaining task is to approximately select the best candidate from $H_{1},\dots,H_{T}$. This is done by implementing a private version of the Scheffé tournament which is itself semi-agnostic, but has a very poor (quadratic) dependence on the number of candidates $T$.

We prove the following result, which gives a semi-agnostic learner whose sample complexity is comparable to that of Theorem 1.1.

Theorem 3.5**.**

Let $\alpha,\beta,\varepsilon\in(0,1)$, and $\zeta>0$ be a constant. Let $\mathcal{H}$ be a set of $m$ distributions and let $P$ be a distribution with $d_{\mathrm{TV}}(P,\mathcal{H})=\operatorname{OPT}$. There is an $\varepsilon$-differentially private algorithm which takes as input $n$ samples from $P$ and with probability at least $1-\beta$, outputs a distribution $\hat{H}\in\mathcal{H}$ with $d_{\mathrm{TV}}(P,\hat{H})\leq 18(3+\zeta)\operatorname{OPT}+\alpha$, as long as

[TABLE]

The running time of the algorithm is $O(m^{2}n\log(1/\alpha)+n\log^{2}(1/\alpha))$.

As discussed above, the algorithm relies on the following variant with a much worse dependence on $m$.

Lemma 3.6**.**

Let $\alpha,\beta,\varepsilon\in(0,1)$. There is an $\varepsilon$-differentially private algorithm which takes as input $n$ samples from $P$ and with probability at least $1-\beta$, outputs a distribution $\hat{H}\in\mathcal{H}$ with $d_{\mathrm{TV}}(P,\hat{H})\leq 9\operatorname{OPT}+\alpha$, as long as

[TABLE]

The running time of the algorithm is $O(m^{2}n)$.

Proof sketch..

We use a different variation of the Scheffé tournament which appears in [DL01]. Non-privately, the algorithm works as follows. For every pair of hypotheses $H,H^{\prime}\in\mathcal{H}$ with Scheffé set $\mathcal{W}_{H,H^{\prime}}=\{x\in\mathcal{X}\mid H(x)>H^{\prime}(x)\}$, let $H(\mathcal{W}_{H,H^{\prime}})$, $H^{\prime}(\mathcal{W}_{H,H^{\prime}})$, and $P(\mathcal{W}_{H,H^{\prime}})$ denote the probability masses of $H,H^{\prime},P$ on $\mathcal{W}_{H,H^{\prime}}$, respectively. Moreover, let $\hat{P}(\mathcal{W}_{H,H^{\prime}})$ denote the fraction of points in the input sample $D$ which lie in $\mathcal{W}_{H,H^{\prime}}$. We declare $H$ to be the winner of the pairwise contest between $H$ and $H^{\prime}$ if $|H(\mathcal{W}_{H,H^{\prime}})-\hat{P}(\mathcal{W}_{H,H^{\prime}})|<|H^{\prime}(\mathcal{W}_{H,H^{\prime}})-\hat{P}(\mathcal{W}_{H,H^{\prime}})|$. Otherwise, we declare $H^{\prime}$ to be the winner. The algorithm outputs the hypothesis $\hat{H}$ which wins the most pairwise contests (breaking ties arbitrarily).

To make this algorithm $\varepsilon$-differentially private, we replace $\hat{P}(\mathcal{W}_{H,H^{\prime}})$ in each pairwise contest with the $(\varepsilon/{m\choose 2})$-differentially private estimate $c_{H,H^{\prime}}=\hat{P}(\mathcal{W}_{H,H^{\prime}})+\operatorname{Lap}({m\choose 2}/\varepsilon n)$. By the composition guarantees of differential privacy, the algorithm as a whole is $\varepsilon$-differentially private.

The analysis of Devroye and Lugosi [DL01, Theorem 6.2] shows that the (private) Scheffé tournament outputs a hypothesis $\hat{H}$ with

[TABLE]

Fix an arbitrary pair $H,H^{\prime}$. A Chernoff bound shows that $|P(\mathcal{W}_{H,H^{\prime}})-\hat{P}(\mathcal{W}_{H,H^{\prime}})|\leq\alpha/32$ with probability at least $1-\beta/(2m^{2})$ as long as $n\geq O(\ln(m/\beta)/\alpha^{2})$. Moreover, properties of the Laplace distribution guarantee $|c_{H,H^{\prime}}-\hat{P}(\mathcal{W}_{H,H^{\prime}})|\leq\alpha/32$ with probability at least $1-\beta/(2m^{2})$ as long as $n\geq O(m^{2}\log(m/\beta)/\alpha\varepsilon)$. The triangle inequality and a union bound over all pairs $H,H^{\prime}$ complete the proof. ∎

Proof of Theorem 3.5.

We now combine the private hypothesis selection algorithm of Theorem 1.1 with the expensive semi-agnostic learner of Lemma 3.6 to prove Theorem 3.5. Define sequences $\alpha_{1}=\alpha/126,\alpha_{2}=2\alpha/126,\dots,\alpha_{T}=2^{T-1}\alpha/126$ and $\varepsilon_{1}=\varepsilon/4,\varepsilon_{2}=\varepsilon/8,\dots,\varepsilon_{T}=2^{-(T+1)}\varepsilon$ for $T=\lceil\log_{2}(1/\alpha)\rceil+1$. For each $t=1,\dots,T$, let $H_{t}$ denote the outcome of a run of Algorithm 2 using accuracy parameter $\alpha_{t}$ and privacy parameter $\varepsilon_{t}$. Finally, use the algorithm of Lemma 3.6 to select a hypothesis from $H_{0},\dots,H_{T}$ using accuracy parameter $\alpha$ and privacy parameter $\varepsilon/2$.

Privacy of this algorithm follows immediately from composition of differential privacy. We now analyze its sample complexity guarantee. By Lemma 3.4, we have that all $T$ runs of Algorithm 2 succeed simultaneously with probability at least $1-\beta/2$ as long as

[TABLE]

Condition on this event occurring. Recall that success of run $t$ of Algorithm 2 means that if $\operatorname{OPT}\in(\alpha_{t-1},\alpha_{t}]$, then $d_{\mathrm{TV}}(P,H_{t})\leq(3+\zeta)\alpha_{t}\leq 2(3+\zeta)\operatorname{OPT}$. Meanwhile, if $\operatorname{OPT}\leq\alpha_{1}=\alpha/126$, then we have $d_{\mathrm{TV}}(P,H_{1})\leq\alpha/18$. Hence, regardless of the value of $\operatorname{OPT}$, there exists a run $t$ such that $d_{\mathrm{TV}}(P,H_{t})\leq 2(3+\zeta)\operatorname{OPT}+\alpha/18$. The algorithm of Lemma 3.6 is now, with probability at least $1-\beta/2$, able to select a hypothesis $\hat{H}$ with $d_{\mathrm{TV}}(P,\hat{H})\leq 9d_{\mathrm{TV}}(P,H_{t})+\alpha/2\leq 18(3+\zeta)\operatorname{OPT}+\alpha$ as long as

[TABLE]

This gives the asserted sample complexity guarantee. ∎

4 An Advanced Method for Private Hypothesis Selection

In Section 3, we provided a simple algorithm whose sample complexity grows logarithmically in the size of the hypothesis class. We now demonstate that this dependence can be improved and, indeed, we can handle infinite hypothesis classes given that their VC dimension is finite and that the cover has small doubling dimension.

To obtain this improved dependence on the hypothesis class size, we must make two improvements to the analysis and algorithm. First, rather than applying a union bound over all the pairwise contests to analyse the tournament, we use a uniform convergence bound in terms of the VC dimension of the Scheffé sets. Second, rather than use the exponential mechanism to select a hypothesis, we use a “GAP-MAX” algorithm [BDRS18]. This takes advantage of the fact that, in many cases, even for infinite hypothesis classes, only a handful of hypotheses will have high scores. The GAP-MAX algorithm need only pay for the hypotheses that are close to optimal. To exploit this, we must move to a relaxation of pure differential privacy which is not subject to strong packing lower bounds (as we describe in Section 5). Specifically, we consider approximate differential privacy, although results with an improved dependence are also possible under various variants of concentrated differential privacy [DR16, BS16, Mir17, BDRS18].

Theorem 4.1**.**

Let $\mathcal{H}$ be a set of probability distributions on $\mathcal{X}$. Let $d$ be the VC dimension of the set of functions $f_{H,H^{\prime}}:\mathcal{X}\to\{0,1\}$ defined by $f_{H,H^{\prime}}(x)=1\iff H(x)>H^{\prime}(x)$ where $H,H^{\prime}\in\mathcal{H}$. There exists a $(\varepsilon,\delta)$-differentially private algorithm which has following guarantee. Let $D=\{X_{1},\dots,X_{n}\}$ be a set of private samples drawn independently from an unknown probability distribution $P$. Let $k=\left|\left\{H\in\mathcal{H}:d_{\mathrm{TV}}(H,P)\leq 7\alpha\right\}\right|$. Suppose there exists a distribution $H^{*}\in\mathcal{H}$ such that $d_{\mathrm{TV}}(P,H^{*})\leq\alpha$. If $n=\Omega\left(\frac{d+\log(1/\beta)}{\alpha^{2}}+\frac{\log(k/\beta)+\min\{\log|\mathcal{H}|,\log(1/\delta)\}}{\alpha\varepsilon}\right)$, then the algorithm will output a distribution $\hat{H}\in\mathcal{H}$ such that $d_{\mathrm{TV}}(P,\hat{H})\leq 7\alpha$ with probability at least $1-\beta$.

Alternatively, we can demand that the algorithm be $\frac{1}{2}\varepsilon^{2}$-concentrated differentially private if $n=\Omega\left(\frac{d+\log(1/\beta)}{\alpha^{2}}+\frac{\log(k/\beta)+\sqrt{\log|\mathcal{H}|}}{\alpha\varepsilon}\right)$.

Comparing Theorem 4.1 to Theorem 1.1, we see that the first (non-private) $\log|\mathcal{H}|$ term is replaced by the VC dimension $d$ and the second (private) $\log|\mathcal{H}|$ term is replaced by $\log k+\log(1/\delta)$. Here $k$ is a measure of the “local” size of the hypothesis class $\mathcal{H}$; its definition is similar to that of the doubling dimension of the hypothesis class under total variation distance.

We note that the $\log(1/\delta)$ term could be large, as the privacy failure probability $\delta$ should be cryptographically small. Thus our result includes statements for pure differential privacy (by using the other term in the minimum with $\delta=0$) and also concentrated differential privacy. Note that, since $d$ and $\log k$ can be upper-bounded by $O(\log|\mathcal{H}|)$, this result supercedes the guarantees of Theorem 1.1.

4.1 VC Dimension

We begin by reviewing the definition of Vapnik-Chervonenkis (VC) dimension and its properties.

Definition 4.2** (VC dimension [VC74]).**

Let $\mathcal{F}$ be a set of functions $f:\mathcal{X}\to\{0,1\}$. The VC dimension of $\mathcal{F}$ is defined to be the largest $d$ such that there exist $x_{1},\cdots,x_{d}\in\mathcal{X}$ and $f_{1},\cdots,f_{2^{d}}\in\mathcal{H}$ such that for all $1\leq i<j\leq 2^{d}$ there exists $1\leq k\leq d$ such that $f_{i}(x_{k})\neq f_{j}(x_{k})$.

For our setting, we must extend the definition of VC dimension from function families to hypothesis classes.

Definition 4.3** (VC dimension of hypothesis class).**

Let $\mathcal{H}$ be a set of probability distributions on a space $\mathcal{X}$. For $H,H^{\prime}\in\mathcal{H}$, define $f_{H,H^{\prime}}:\mathcal{X}\to\{0,1\}$ by $f(x)=1\iff H(x)>H^{\prime}(x)$. Define $\mathcal{F}(\mathcal{H})=\{f_{H,H^{\prime}}:H,H^{\prime}\in\mathcal{H}\}$. We define the VC dimension of $\mathcal{H}$ to be the VC dimension of $\mathcal{F}(\mathcal{H})$.555Here, for simplicity, we assume that each distribution $H$ is given by a density function $H(\cdot)$. More generally, we define the VC dimension of $\mathcal{H}$ to be the smallest $d$ such that there exists a function family $\mathcal{F}\subseteq\{0,1\}^{\mathcal{X}}$ of VC dimension $d$ with the property that, for all $H,H^{\prime}\in\mathcal{H}$ we have $d_{\mathrm{TV}}(H,H^{\prime})=\sup_{f\in\mathcal{F}}\mbox{\bf E}_{X\leftarrow H}\!\left[f(X)\right]-\mbox{\bf E}_{X\leftarrow H^{\prime}}\!\left[f(X)\right]$, where the supremum is over $f$ measurable with respect to both $H$ and $H^{\prime}$. We ignore this technicality throughout.

The key property of VC dimension is the following uniform convergence bound, which we use in place of a union bound.

Theorem 4.4** (Uniform Convergence [Tal94]).**

Let $\mathcal{F}$ be a set of functions $f:\mathcal{X}\to\{0,1\}$ with VC dimension $d$. Let $P$ be a distribution on $\mathcal{X}$. Then

[TABLE]

whenever $n=\Omega\left(\frac{d+\log(1/\beta)}{\alpha^{2}}\right)$. Here $f(D):=\frac{1}{n}\sum_{x\in D}f(x)$ and $f(P):=\mbox{\bf E}_{X\leftarrow P}\!\left[f(X)\right]$.

It is immediate from Definition 4.2 that $VC(\mathcal{F})\leq\lfloor\log_{2}|\mathcal{F}|\rfloor$. Thus Theorem 4.4 subsumes the union bound used in the proof of Theorem 1.1.

The relevant application of uniform convergence for our algorithm is the following lemma (roughly the equivalent of Lemma 3.1), which says that good hypotheses have high scores, and bad hypotheses have low scores.

Lemma 4.5**.**

Let $\mathcal{H}$ be a collection of probability distributions on $\mathcal{X}$ with VC dimension $d$.

Let $S:\mathcal{H}\times\mathcal{X}^{n}\to\mathbb{R}$ be a score function similar to (1), namely

[TABLE]

where $\mathbb{I}$ denotes the indicator function.

Let $P$ be a distribution on $\mathcal{X}$. Let $\alpha,\beta>0$ and $n\geq O(\frac{1}{\alpha^{2}}(d+\log(1/\beta)))$. Suppose there exists $H^{*}\in\mathcal{H}$ with $d_{\mathrm{TV}}(P,H^{*})\leq\alpha$. Then, with probability at least $1-\beta$ over $D\leftarrow P^{n}$, we have

• •

$S(H^{*},D)>\alpha n$* and*

• •

$S(H,D)=0$* for all $H\in\mathcal{H}$ with $d_{\mathrm{TV}}(H,P)>7\alpha$.*

Proof.

For $H,H^{\prime}\in\mathcal{H}$, define $f_{H,H^{\prime}}:\mathcal{X}\to\{0,1\}$ by $f_{H,H^{\prime}}(x)=1\iff H(x)>H^{\prime}(x)$. Note that $|\{x\in D:H(x)>H^{\prime}(x)\}|=\sum_{x\in D}f_{H,H^{\prime}}(x)$ and $d$ is the VC dimension of the function class $\{f_{H,H^{\prime}}:H,H^{\prime}\in\mathcal{H}\}$. By Theorem 4.4, if $n=\Omega\left(\frac{d+\log(1/\beta)}{\alpha^{2}}\right)$, then

[TABLE]

We condition on this event happening.

In order to prove the first conclusion – namely, $S(H^{*},D)>\alpha n$ – it remains to show that, for all $H^{\prime}\in\mathcal{H}$, we have either $d_{\mathrm{TV}}(H^{*},H^{\prime})\leq 6\alpha$ or

[TABLE]

If $d_{\mathrm{TV}}(H^{*},H^{\prime})\leq 6\alpha$, we are done, so assume $d_{\mathrm{TV}}(H^{*},H^{\prime})>6\alpha$. By the uniform convergence event we have conditioned on,

[TABLE]

from which the desired conclusion follows.

In order to prove the second conclusion – namely, $S(H,D)=0$ for all $H\in\mathcal{H}$ with $d_{\mathrm{TV}}(H,P)>7\alpha$ – it suffices to show that one $H^{\prime}\in\mathcal{H}$ yields a score of zero for any $H\in\mathcal{H}$ with $d_{\mathrm{TV}}(H,P)>7\alpha$. In particular, we show that $H^{\prime}=H^{*}$ yields a score of zero for any such $H$. That is, if $d_{\mathrm{TV}}(H,P)>7\alpha$, then $d_{\mathrm{TV}}(H,H^{*})>6\alpha$ and

[TABLE]

By the triangle inequality $d_{\mathrm{TV}}(H,H^{*})\geq d_{\mathrm{TV}}(H,P)-d_{\mathrm{TV}}(P,H^{*})>7\alpha-\alpha=6\alpha$, as required. By the uniform convergence event we have conditioned on,

[TABLE]

which completes the proof. ∎

4.2 GAP-MAX Algorithm

In place of the exponential mechanism for privately selecting a hypothesis we use the following algorithm that works under a “gap” assumption. That is, we assume that there is a $5\alpha n$ gap between the highest score and the $(k+1)$-th highest score. Rather than paying in sample complexity for the total number of hypotheses we pay for the number of high-scoring hypotheses $k$.

This algorithm is based on the GAP-MAX algorithm of Bun, Dwork, Rothblum, and Steinke [BDRS18]. However, we combine their GAP-MAX algorithm with the exponential mechanism to improve the dependence on the parameter $k$.

Theorem 4.6**.**

Let $\mathcal{H}$ and $\mathcal{X}$ be arbitrary sets. Let $S:\mathcal{H}\times\mathcal{X}^{n}\to\mathbb{R}$ have sensitivity at most 1 in its second argument – that is, for all $H\in\mathcal{H}$ and all $D,D^{\prime}\in\mathcal{X}^{n}$ differing in a single example, $|S(H,D)-S(H,D^{\prime})|\leq 1$.

For $D\in\mathcal{X}^{n}$ and $\alpha>0$, define

[TABLE]

Given parameters $\varepsilon,\delta,\beta>0$ and $n,k\geq 1$, there exists a $(\varepsilon,\delta)$-differentially private randomized algorithm $M:\mathcal{X}^{n}\to\mathcal{H}$ such that, for all $D\in\mathcal{X}^{n}$ and all $\alpha>0$,

[TABLE]

provided $n=\Omega\left(\frac{\min\{\log|\mathcal{H}|,\log(1/\delta)\}+\log(k/\beta)}{\alpha\varepsilon}\right)$.

Furthermore, given $\varepsilon,\beta>0$ and $n,k\geq 1$, there exists a $\frac{1}{2}\varepsilon^{2}$-concentrated differentially private [BS16] algorithm $M:\mathcal{X}^{n}\to\mathcal{H}$ such that, for all $D\in\mathcal{X}^{n}$ and all $\alpha>0$,

[TABLE]

provided $n=\Omega\left(\frac{\sqrt{\log|\mathcal{H}|}+\log(k/\beta)}{\alpha\varepsilon}\right)$.

Proof.

We begin by describing the algorithm.

Let $m=\left\lceil\frac{k^{2}}{\beta}\right\rceil$ and let $G:\mathcal{H}\to[m]$ be a uniformly random function.666It suffices for $G$ to be a drawn from a universal hash function family. 2. 2.

Randomly select $B\in[m]$ with

[TABLE] 3. 3.

Define $\mathcal{H}_{B}=\{H\in\mathcal{H}:G(H)=B\}$. Let $H_{B}^{1}=\operatorname{argmax}_{H\in\mathcal{H}_{B}}S(H,D)$ and $H_{B}^{2}=\operatorname{argmax}_{H\in\mathcal{H}_{B}\setminus\{H_{B}^{1}\}}S(H,D)$, breaking ties arbitrarily. (That is, $\mathcal{H}_{B}$ is the $B$-th “bin” and $H_{B}^{1}$ and $H_{B}^{2}$ are the items in this bin with the largest and second-largest scores respectively.) Define $S_{B}^{\prime}:\mathcal{H}_{B}\times\mathcal{X}^{n}\to\mathbb{R}$ by

[TABLE]

(Note that $S_{B}^{\prime}$ has sensitivity 1 and $S_{B}^{\prime}(H,D)=0$ whenever $H\neq H_{B}^{1}$.) 4. 4.

Let $\mathcal{D}$ be a distribution on $\mathbb{R}$ such that adding a sample from $\mathcal{D}$ to a sensitivity-$1$ function provides $(\varepsilon/4,\delta/2)$-differential privacy (or, respectively, $\frac{1}{6}\varepsilon^{2}$-concentrated differential privacy). For example, $\mathcal{D}$ could be a Laplace distribution with scale $4/\varepsilon$ truncated to the interval $[-t,t]$ for $t=4(1+\log(1/\delta))/\varepsilon$ (or unbounded if $\delta=0$). To attain concentrated differential privacy, we can set $\mathcal{D}=N\left(0,\frac{3}{\varepsilon^{2}}\right)$, a centered Gaussian with variance $3/\varepsilon^{2}$. 5. 5.

Draw a sample $Z_{H}$ i.i.d. from $\mathcal{D}$ corresponding to every $H\in\mathcal{H}_{B}$. 6. 6.

Return $H^{*}=\operatorname{argmax}_{H\in\mathcal{H}_{B}}S_{B}^{\prime}(H,D)+Z_{H}$.

The selection of $B$ is an instantiation of the exponential mechanism [MT07] and is $(\varepsilon/2,0)$-differentially private. The selection of $H^{*}$ in the final step is a GAP-MAX algorithm [BDRS18] and is $(\varepsilon/2,\delta)$-differentially private. By composition, the entire algorithm is $(\varepsilon,\delta)$-differentially private (or, respectively, $\frac{1}{2}\varepsilon^{2}$-concentrated differentially private).

For the utility analysis, in order for the algorithm to output a good $H^{*}$, it suffices for the following three events to occur.

• •

$S(H_{B}^{1},D)\geq\sup_{H^{\prime}\in\mathcal{H}}S(H^{\prime},D)-\alpha n$.

That is, restricting the search to $\mathcal{H}_{B}$, rather than all of $\mathcal{H}$, only reduces the score of the optimal choice by $\alpha n$. The exponential mechanism ensures that this happens with probability at least $1-\beta/4$, as long as $n\geq\frac{4\log(2k/\beta)}{\varepsilon\alpha}$.

• •

$S(H_{B}^{2},D)<\sup_{H^{\prime}\in\mathcal{H}}S(H^{\prime},D)-5\alpha n$.

That is, the second-highest score within $\mathcal{H}_{B}$ is at least $5\alpha n$ less than the highest score overall. We have assumed that there are at most $k$ elements $H\in\mathcal{H}$ such that $S(H,D)\geq\sup_{H^{\prime}\in\mathcal{H}}S(H^{\prime},D)-5\alpha n$. Call these “large elements.” Since $G:\mathcal{H}\to[m]$ is random and $m\geq k^{2}/\beta$, the probability that an arbitrary but fixed pair of large elements collide – that is, are in the same $H_{B}$ is $1/m\leq\beta/k^{2}$. If we union bound over the ${k\choose 2}<k^{2}/2$ pairs, we see that the probability of any collisions is at most $\beta/2$. Thus, the probability that more than one large element satisfies $G(H)=B$ is at most $\beta/2$. This suffices for the event to occur.

• •

$\sup_{H\in\mathcal{H}_{B}}|Z_{H}|\leq\alpha n$.

If the noise distribution $\mathcal{D}$ is supported on $[-\alpha n,\alpha n]$, then this condition holds with probability $1$. For the truncated Laplace distribution, this is possible whenever $n\geq 1+4\log(1/\delta)/\alpha\varepsilon$. Alternatively, we can use unbounded Laplace noise and a union bound to show that this event occurs with probability at least $1-\beta/4$ whenever $n\geq 4\log(4|\mathcal{H}_{B}|/\beta)/\varepsilon\alpha$. For Gaussian noise, $n\geq\frac{3}{\varepsilon\alpha}\sqrt{\log(4|\mathcal{H}_{B}|/\beta)}$ suffices.

Assuming the first and second events occur, we have $S^{\prime}_{B}(H_{B}^{1},D)=\frac{S(H_{B}^{1},D)-S(H_{B}^{2},D)}{2}>2\alpha n$. Given this, the third event implies $H^{*}=H_{B}^{1}$. Finally, the first event then implies $S(H^{*},D)\geq\sup_{H^{\prime}\in\mathcal{H}}S(H^{\prime},D)-\alpha n$, as required. A union bound over the three events completes the proof. ∎

Now we can combine the VC-based uniform convergence bound with the GAP-MAX algorithm to prove our result.

Proof of Theorem 4.1.

By Lemma 4.5, with high probability over the draw of the dataset $D$, our score function satisfies $\sup_{H\in\mathcal{H}}S(H,D)\geq S(H^{*},D)>\alpha n$ and $S(H,D)=0$ whenever $d_{\mathrm{TV}}(H,P)>7\alpha$. This requires $n=\Omega(d/\alpha^{2})$.

Note that the score function $S$ has sensitivity-1, since it is the supremum of counts. Conditioned on the uniform convergence event, the maximum score is at least $\alpha n$ and there are at most $k$ elements of $\mathcal{H}$ with score greater than [math]. Thus we can apply the GAP-MAX algorithm of Theorem 4.6. If $n=\Omega((\min\{\log|\mathcal{H}|,\log(1/\delta)\}+\log(k))/\alpha\varepsilon)$, then with high probability, the algorithm outputs $\hat{H}\in\mathcal{H}$ with score at least $\frac{4}{5}\alpha n$, as required. ∎

5 Packings, Lower Bounds, and Relations to Covers

In this section, we show that the sample complexity of our algorithms for private hypothesis selection with pure differential privacy cannot be improved, at least for constant values of the proximity parameter $\alpha$. We first apply a packing argument [HT10, BBKN14] to show a lower bound which is logarithmic in the packing number of the class of distributions (Lemma 5.1). We then state a folklore relationship between the sizes of maximal packings and minimal covers (Lemma 5.2), which shows that instantiating our private hypothesis selection algorithm with a minimal cover gives essentially optimal sample complexity (Theorem 5.3).

Lemma 5.1**.**

Suppose there exists an $\alpha$-packing $\mathcal{P}_{\alpha}$ of a set of distributions $\mathcal{H}$. Then any $\varepsilon$-differentially private algorithm which takes as input samples $X_{1},\dots,X_{n}\sim P$ for some $P\in\mathcal{H}$ and produces a distribution $\hat{H}$ such that $d_{\mathrm{TV}}(P,\hat{H})\leq\alpha/2$ with probability $\geq 9/10$ requires

[TABLE]

One might conjecture a stronger version of this lemma exists, and that one could prove the lower bound $n=\Omega\left(\frac{\log|\mathcal{P}_{\alpha}|}{\alpha\varepsilon}\right)$. However, such a statement cannot be true in general. For example, consider an $\alpha$-packing of $N(\mu,1)$ where $\mu\in[-R,R]$, which would have size $\Omega(R/\alpha)$. If such a lemma were true, it would imply a lower bound of $\tilde{\Omega}\left(\frac{\log R}{\alpha\varepsilon}\right)$, which contradicts known upper bounds. One way to prove a lower bound achieving such a dependence on $\alpha$ would be to have a single “central” distribution which is close to all distributions in the packing (see an argument of this sort in Theorem 5.13 of [Vad17]). However, we do not explore this here, as our goal is to match our upper bound which is stated in terms of a generic cover.

Proof.

Let $M$ be a $\varepsilon$-differentially private algorithm with the stated accuracy requirement, and denote by $M(P^{n})$ the distribution on hypotheses obtained by running $M$ on $n$ i.i.d. samples from a distribution $P\in\mathcal{H}$. For each $P\in\mathcal{P}_{\alpha}$, let $B_{P}$ denote the set of distributions which are at total variation distance at most $\alpha/2$ from $P$. Then the accuracy requirement implies that $\mathrm{Pr}_{\hat{H}\leftarrow M(P^{n})}\!\left[\hat{H}\in B_{P}\right]\geq 9/10$ for all $P\in\mathcal{H}$. Let $P_{0}\in\mathcal{P}_{\alpha}$ be an arbitrary packing element. Note that, trivially, samples from $P^{n}$ and $P^{n}_{0}$ have Hamming distance at most $n$ for any $P$. Recall the group privacy property of differential privacy, which states that if $M$ is $\varepsilon$-DP, then $\Pr[M(X)\in S]\leq\exp\left(\varepsilon d(X,X^{\prime})\right)\cdot\Pr[M(X^{\prime})\in S]$ for any set $S\subseteq\mathrm{Range}(M)$, where $d(X,X^{\prime})$ is the Hamming distance between the two datasets. Applying this property with $P^{n}$ and $P^{n}_{0}$, we have

[TABLE]

for every $P\in\mathcal{P}_{\alpha}$. The fact that $\mathcal{P}_{\alpha}$ is an $\alpha$-packing implies that the sets $B_{P}$ are all disjoint, and hence

[TABLE]

Rearranging gives us the stated lower bound on $n$. ∎

The following lemma is a well-known folklore relationship between packing and covering numbers. We include a proof for completeness.

Lemma 5.2**.**

For a set of distributions $\mathcal{H}$, let $p_{\alpha}$ and $c_{\alpha}$ be the size of the largest $\alpha$-packing and smallest $\alpha$-cover of $\mathcal{H}$, respectively. Then

[TABLE]

Proof.

We first prove the inequality on the left. Let $\mathcal{C}_{\alpha}$ be a cover of $\mathcal{H}$ of minimal size $c_{\alpha}$. If $c_{\alpha}=\infty$, we are done. Otherwise, let $S\subseteq\mathcal{H}$ be any set of distributions of size at least $c_{\alpha}+1$. By the pigeonhole principle, there exists $P\in\mathcal{C}_{\alpha}$ and two distinct distributions $Q,Q^{\prime}\in S$ such that $d_{\mathrm{TV}}(P,Q)\leq\alpha$ and $d_{\mathrm{TV}}(P,Q^{\prime})\leq\alpha$. Hence $d_{\mathrm{TV}}(Q,Q^{\prime})\leq 2\alpha$ by the triangle inequality, so $S$ cannot be $(2\alpha)$-packing of $\mathcal{H}$. This suffices to show that $p_{2\alpha}\leq c_{\alpha}$.

Next, we prove the inequality on the right. Let $\mathcal{P}_{\alpha}$ be a maximal $\alpha$-packing with size $|\mathcal{P}_{\alpha}|=p_{\alpha}$. If $p_{\alpha}=\infty$, we are done. Otherwise, we claim that $\mathcal{P}_{\alpha}$ is also an $\alpha$-cover of $\mathcal{H}$, and hence $c_{\alpha}\leq|\mathcal{P}_{\alpha}|=p_{\alpha}$. To see this, suppose for the sake of contradiction that there were a distribution $P\in\mathcal{H}$ with $d_{\mathrm{TV}}(P,\mathcal{P}_{\alpha})>\alpha$. Then we could add $P$ to $\mathcal{P}_{\alpha}$ to produce a strictly larger packing, contradicting the maximality of $\mathcal{P}_{\alpha}$. ∎

Theorem 5.3**.**

Let $\mathcal{H}$ be a set of distributions, and let $n^{*}_{\alpha}$ denote the minimum number of samples such that there exists an $\varepsilon$-differentially private algorithm which takes as input samples $X_{1},\dots,X_{n^{*}_{\alpha}}\sim P$ for an arbitrary $P\in\mathcal{H}$ and outputs a distribution $\hat{H}$ such that $d_{\mathrm{TV}}(P,\hat{H})\leq\alpha/2$ with probability $\geq 9/10$. Then there exists a cover of $\mathcal{H}$ such that the instantiation of the algorithm underlying Theorem 1.1 with this cover takes as input $n=\Omega(n^{*}_{\alpha}\cdot(\varepsilon/\alpha^{2}+1/\alpha))$ samples from an arbitrary $P\in\mathcal{H}$ and outputs a $\hat{H}$ such that $d_{\mathrm{TV}}(P,\hat{H})\leq(6+2\zeta)\alpha$ with probability $\geq 9/10$ for any constant $\zeta>0$.

Proof.

Let $p_{\alpha}$ denote the size of the largest $\alpha$-packing of $\mathcal{H}$. By Lemma 5.1, we have $n^{*}_{\alpha}=\Omega(\log p_{\alpha}/\varepsilon)$. On the other hand, by Lemma 5.2, we know that there exists an $\alpha$-cover $\mathcal{C}_{\alpha}$ of $\mathcal{H}$ with $|\mathcal{C}_{\alpha}|\leq p_{\alpha}$. Hence $\log|\mathcal{C}_{\alpha}|\leq O(\varepsilon\cdot n^{*}_{\alpha})$ and the asserted sample complexity guarantee follows from Corollary 1.2. ∎

6 Applications of Hypothesis Selection

In this section, we give a number of applications of Theorem 1.1, primarily to obtain sample complexity bounds for learning a number of distribution classes of interest. Recall Corollary 1.2, which is an immediate corollary of Theorem 1.1. This indicates that we can privately semi-agnostically learn a class of distributions with a number of samples proportional to the logarithm of its covering number. See 1.2

Note that the factor of $(6+2\zeta)\alpha$ in the corollary statement (versus $(3+\zeta)\alpha$ in the statement of Theorem 1.1) is due to the fact the algorithm is semi-agnostic, and the closest element in the cover is $2\alpha$-close to $P$, rather than just $\alpha$-close.

We instantiate this result to give the sample complexity results for semi-agnostically learning product distributions (Section 6.1), Gaussian distributions (Section 6.2), sums of some independent random variable classes (Section 6.3), piecewise polynomials (Section 6.4), and mixtures (Section 6.5). Furthermore, we mention an application to private PAC learning (Section 6.6), when the distribution of unlabeled examples is known to come from some hypothesis class.

6.1 Product Distributions

As a first application, we first give an $\varepsilon$-differentially private algorithm for learning product distributions over discrete alphabets.

Definition 6.1**.**

A $(k,d)$-product distribution is a distribution over $[k]^{d}$, such that its marginal distributions are independent (i.e., the distribution is the product of its marginals).

We start by constructing a cover for product distributions.

Lemma 6.2**.**

There exists an $\alpha$-cover of the set of $(k,d)$-product distributions of size

[TABLE]

Proof.

Consider some fixed product distribution $P$, with marginal distributions $(P_{1},\dots,P_{d})$. We will construct a cover that contains a distribution $Q$ (with marginals $(Q_{1},\dots,Q_{d})$) that is $\alpha$-close in total variation distance.

First, by triangle inequality, we have that $d_{\mathrm{TV}}(P,Q)\leq\sum_{i=1}^{d}d_{\mathrm{TV}}(P_{i},Q_{i})$, so it suffices to approximate each marginal distribution to accuracy $\alpha/d$. Stated another way, we must generate an $(\alpha/d)$-cover of distributions over $[k]$, and we can then take its $d$-wise Cartesian product. Raising the size of this underlying cover to the power $d$ gives us the size of the overall cover.

To $(\alpha/d)$-cover a distribution over $[k]$, we will additively grid the probability of each symbol at granularity $\Theta\left(\frac{\alpha}{kd}\right)$, choosing the probability of the last symbol $k$ such that the sum is normalized. This will incur $\Theta\left(\frac{\alpha}{kd}\right)$ error per symbol (besides for symbol $k$), and summing over the $k-1$ symbols accumulates error $\Theta\left(\frac{\alpha}{d}\right)$. It can also be argued that the error on symbol $k$ is $O\left(\frac{\alpha}{d}\right)$ – with an appropriate choice of granularity, this gives us an $(\alpha/d)$-cover for distributions over $[k]$. The size of this cover is $O\left(\frac{kd}{\alpha}\right)^{k-1}$, which allows us to conclude the lemma statement. ∎

With this cover in hand, applying Corollary 1.2 allows us to conclude the following sample complexity upper bound.

Corollary 6.3**.**

Suppose we are given a set of samples $X_{1},\dots,X_{n}\sim P$, where $P$ is $\alpha$-close to a $(k,d)$-product distribution. Then for any constant $\zeta>0$, there exists an $\varepsilon$-differentially private algorithm which outputs a $(k,d)$-product distribution $H^{*}$ such that $d_{\mathrm{TV}}(P,H^{*})\leq(6+2\zeta)\alpha$ with probability $\geq 9/10$, so long as

[TABLE]

This gives the first $\tilde{O}(d)$ sample algorithm for learning a binary product distribution in total variation distance under pure differential privacy, improving upon the work of Kamath, Li, Singhal, and Ullman [KLSU19] by strengthening the privacy guarantee at a minimal cost in the sample complexity. The natural way to adapt their result from concentrated to pure differential privacy would require $\Omega(d^{3/2})$ samples.

Remark 6.4**.**

Properly learning a product distribution over $\{0,1\}^{d}$ to total variation distance $\leq\frac{1}{2}$ implies learning its mean $\mu\in[0,1]^{d}$ up to $\ell_{1}$ error $\leq 2\sqrt{d}$; see Lemma 6.5 below.

Thus Corollary 6.3 implies a $\varepsilon$-differentially private algorithm which takes $n=\tilde{O}(d/\varepsilon)$ samples from a product distribution $P$ on $\{0,1\}^{d}$ and, with high probability, outputs an estimate $\hat{\mu}$ of its mean $\mu$ with $\|\hat{\mu}-\mu\|_{1}\leq 2\sqrt{d}$.

In contrast, for non-product distributions over the hypercube, estimating the mean to the same accuracy under $\varepsilon$-differential privacy requires $n=\Omega(d^{3/2}/\varepsilon)$ samples [HT10, SU15]. Thus we have a polynomial separation between estimating product and non-product distributions under pure differential privacy.

Lemma 6.5**.**

If $P$ and $Q$ are product distributions on $\mathbb{R}^{d}$ with $d_{\mathrm{TV}}(P,Q)\leq\frac{1}{2}$ and per-coordinate variance at most $\sigma^{2}$, then

[TABLE]

Proof.

Let $\mu=\mbox{\bf E}_{X\leftarrow P}\!\left[X\right]\in\mathbb{R}^{d}$ and $\mu^{\prime}=\mbox{\bf E}_{X\leftarrow Q}\!\left[X\right]\in\mathbb{R}^{d}$. Let $\tau=\|\mu-\mu^{\prime}\|_{1}$. Let $\nu=\mathrm{sign}(\mu-\mu^{\prime})\in\{-1,+1\}^{d}$ so that $\langle\nu,\mu-\mu^{\prime}\rangle=\tau$. We have

[TABLE]

Rearranging yields $\tau\leq 4\sqrt{d\sigma^{2}}$, as required. ∎

6.2 Gaussian Distributions

We next give private algorithms for learning Gaussian distributions.

Definition 6.6**.**

A Gaussian distribution $\mathcal{N}(\mu,\Sigma)$ in $\mathbb{R}^{d}$ is a distribution with PDF

[TABLE]

We describe covers for Gaussian distributions with known and unknown covariance.

Lemma 6.7**.**

There exists an $\alpha$-cover of the set of Gaussian distributions $\mathcal{N}(\mu,I)$ in $d$ dimensions with $\|\mu\|_{2}\leq R$ of size

[TABLE]

Proof.

It is well-known that estimating a Gaussian distribution with unknown mean in total variation distance corresponds to estimating $\mu$ in $\ell_{2}$-distance (see, e.g., [DKK*+*16]). By the triangle inequality, in order to $\alpha$-cover the space, it suffices to $(\alpha/d)$-cover each standard basis direction. Since we know the mean in each direction is bounded by $R$, a simple additive grid in each direction with granularity $\Theta\left(\frac{\alpha}{d}\right)$ will suffice, resulting in a cover for each direction of size $O\left(\frac{dR}{\alpha}\right)$. Taking the Cartesian product over $d$ dimensions gives the desired result. ∎

Lemma 6.8**.**

There exists an $\alpha$-cover of the set of Gaussian distributions $\mathcal{N}(\mu,\Sigma)$ in $d$-dimensions with $\|\mu\|_{2}\leq R$ and $I\preceq\Sigma\preceq\kappa I$ of size

[TABLE]

Proof.

The former term is obtained similarly to the expression in Lemma 6.7. Since $I\preceq\Sigma$, we can still bound the total variation contribution by the $\ell_{2}$-distance between the mean vectors. We thus turn our attention to the latter term. To construct our cover, we must argue about the total variation distance between $\mathcal{N}(0,\Sigma)$ and $\mathcal{N}(0,\hat{\Sigma})$. If $|\Sigma(i,j)-\hat{\Sigma}(i,j)|\leq\gamma$, and $I\preceq\Sigma$, Proposition 32 of [VV10] implies:

[TABLE]

We will thus perform a gridding, in order to approximate each entry of $\Sigma$ to an additive $O(\gamma)=O(\alpha/d)$. However, in order to ensure that the resulting matrix is PSD, we grid over entries of $\hat{\Sigma}$’s Cholesky decomposition, rather than grid for $\hat{\Sigma}$ itself. Since the largest element of $\Sigma$ is bounded by $\kappa$, the larest element of its Cholesky decomposition must be bounded by $\sqrt{\kappa}$. An additive grid over the range $[0,\sqrt{\kappa}]$ with granularity $O(\gamma/\sqrt{\kappa})$ suffices to get $\hat{\Sigma}$ which bounds the entrywise distance as $O(\gamma)$. This requires $O(d\kappa/\alpha)$ candidates per entry, and we take the Cartesian product over all $d(d+1)/2$ entries of the Cholesky decomposition, giving the desired result. ∎

In addition, we can obtain bounds of the VC dimension of the Scheffé sets of Gaussian distributions.

Lemma 6.9**.**

The set of Gaussian distributions with fixed variance – i.e., all $\mathcal{N}(\mu,I)$ with $\mu\in\mathbb{R}^{d}$ – has VC dimension $d+1$. Furthermore, the set of Gaussians with unknown variance – i.e., all $\mathcal{N}(\mu,\Sigma)$ with $\mu\in\mathbb{R}^{d}$ and $\Sigma\in\mathbb{R}^{d\times d}$ positive definite – has VC dimension $O(d^{2})$.

Proof.

For Gaussians with fixed variance, the Scheffé sets correspond to linear threshold functions, which have VC dimension $d+1$. For Gaussians with unknown variance, the Scheffé sets correspond to quadratic threshold functions, which have VC dimension ${d+2\choose 2}=O(d^{2})$ [Ant95]. ∎

Combining the covers of Lemmas 6.7 and 6.8 and the VC bound of Lemma 6.9 with Theorem 4.1 implies the following corollaries for Gaussian estimation.

Corollary 6.10**.**

Suppose we are given a set of samples $X_{1},\dots,X_{n}\sim P$, where $P$ is $\alpha$-close to a Gaussian distribution $\mathcal{N}(\mu,I)$ in $d$-dimensions with $\|\mu\|\leq R$. Then for any constant $\zeta>0$, there exists an $\varepsilon$-differentially private algorithm which outputs a Gaussian distribution $H^{*}$ such that $d_{\mathrm{TV}}(P,H^{*})\leq(6+2\zeta)\alpha$ with probability $\geq 9/10$, so long as

[TABLE]

Corollary 6.11**.**

Suppose we are given a set of samples $X_{1},\dots,X_{n}\sim P$, where $P$ is $\alpha$-close to a Gaussian distribution $\mathcal{N}(\mu,\Sigma)$ in $d$-dimensions with $\|\mu\|\leq R$ and $I\preceq\Sigma\preceq\kappa I$. Then for any constant $\zeta>0$, there exists an $\varepsilon$-differentially private algorithm which outputs a Gaussian distribution $H^{*}$ such that $d_{\mathrm{TV}}(P,H^{*})\leq(6+2\zeta)\alpha$ with probability $\geq 9/10$, so long as

[TABLE]

Similar to the product distribution case, these are the first $\tilde{O}(d)$ and $\tilde{O}(d^{2})$ sample algorithms for learning Gaussians total variation distance under pure differential privacy, improving upon the concentrated differential privacy results of Kamath, Li, Singhal, and Ullman [KLSU19].

6.2.1 Gaussians with Unbounded Mean

Extending Corollary 6.10, we consider multivariate Gaussian hypotheses with known covariance and unknown mean, without assuming bound on the mean (the parameter $R$ in the discussion above). To handle the unbounded mean we must relax to approximate differential privacy.

In place of Lemma 6.7, we construct a locally small cover:

Lemma 6.12**.**

For any $d\in\mathbb{N}$ and $\alpha\in(0,1/30]$, there exists an $\alpha$-cover $\mathcal{C}_{\alpha}$ of the set of Gaussian distributions $\mathcal{N}(\mu,I)$ in $d$ dimensions satisfying

[TABLE]

Proof.

For $\mu,\mu^{\prime}\in\mathbb{R}^{d}$, we have

[TABLE]

Furthermore, for any $c>0$,

[TABLE]

Let

[TABLE]

Fix $\mu\in\mathbb{R}^{d}$. Let $\mu^{*}=\mu\frac{\sqrt{d}}{\alpha\sqrt{8\pi}}\in\mathbb{R}^{d}$ and let $m\in\mathbb{Z}^{d}$ be $\mu^{*}$ rounded to the nearest integer coordinate-wise, so that $\|m-\mu^{*}\|_{\infty}\leq\frac{1}{2}$. Then

[TABLE]

since $\|\mu^{*}-m\|_{2}\leq\sqrt{d}\|\mu^{*}-m\|_{\infty}\leq\frac{\sqrt{d}}{2}$. This proves that $\mathcal{C}_{\alpha}$ is a $\alpha$-cover of $\{\mathcal{N}(\mu,I):\mu\in\mathbb{R}^{d}\}$.

It remains to show that the cover is “locally small”. Let $m^{\prime}\in\mathbb{Z}^{d}$. Then

[TABLE]

where the final inequality follows by setting $c=30\alpha\leq 1$. Thus

[TABLE]

Now we note that any $w\in\mathbb{Z}^{d}$ with $\|w\|_{1}\leq r$ can be written as $w=x-y$ where $x,y\in\mathbb{Z}^{d}$ with $\sum_{i=1}^{d}x_{i}+y_{i}=r$ and, for all $i\in[d]$, we have $x_{i}\geq 0$ and $y_{i}\geq 0$. Instead of counting these $w$ vectors, we can count such $(x,y)$ vector pairs. We can interpret a pair of $x,y$ vectors as a way of putting $r$ balls into $2d$ bins or $r$ “stars” and $2d-1$ “bars”. We can thus count

[TABLE]

∎

Applying Theorem 4.1 with the cover of Lemma 6.12 and the VC bound from Lemma 6.9 now yields an algorithm.

Corollary 6.13**.**

Suppose we are given a set of samples $X_{1},\dots,X_{n}\sim P$, where $P$ is a spherical Gaussian distribution $\mathcal{N}(\mu,I)$ in $d$-dimensions. Then there exists a $(\varepsilon,\delta)$-differentially private algorithm which outputs a spherical Gaussian distribution $H^{*}$ such that $d_{\mathrm{TV}}(P,H^{*})\leq 7\alpha$ with probability $\geq 1-2^{-d}$, so long as

[TABLE]

Karwa and Vadhan [KV18] give an algorithm for estimating a univariate Gaussian with unbounded mean. One can consider applying their algorithm independently to the $d$ coordinates (which is done in [KLSU19]), giving a sample complexity bound of $\tilde{O}\left(\frac{d}{\alpha^{2}}+\frac{d}{\alpha\varepsilon}+\frac{\sqrt{d}\log^{3/2}(1/\delta)}{\varepsilon}\right)$, which our bound dominates except for very small values of $\alpha$.

6.2.2 Univariate Gaussians with Unbounded Mean and Variance

Our methods also allow us to derive learning algorithms for univariate Gaussians with unknown mean and variance.

Lemma 6.14**.**

For all $\alpha$ less then some absolute constant, there exists an $\alpha$-cover $\mathcal{C}_{\alpha}$ of the set of univariate Gaussian distributions satisfying

[TABLE]

Proof.

For all $\mu,\tilde{\mu}\in\mathbb{R}$ and all $\sigma,\tilde{\sigma}>0$, we have [DMR18, Thm 1.3]

[TABLE]

Let $\beta=\alpha$ and $\gamma=\log(1+\alpha/2)$. Define the set of distributions

[TABLE]

We first show that $\mathcal{C}_{\alpha}$ is an $\alpha$-cover: Let $\mu\in\mathbb{R}$ and $\sigma>0$. Let $n=\left[\frac{\log\sigma}{\gamma}\right]$ and $m=\left[\frac{\mu}{\beta e^{\gamma n}}\right]$, where $[x]$ denotes the nearest integer to $x$, satisfying $|x-[x]|\leq\frac{1}{2}$. Let $\tilde{\sigma}=e^{\gamma n}$ and $\tilde{\mu}=\beta e^{\gamma n}m$ so that $e^{-\gamma}\leq\frac{\tilde{\sigma}^{2}}{\sigma^{2}}\leq e^{\gamma}$ and $|\mu-\tilde{\mu}|\leq\frac{1}{2}\beta e^{\gamma n}=\frac{1}{2}\beta\tilde{\sigma}$. Thus $\mathcal{N}(\tilde{\mu},\tilde{\sigma}^{2})\in\mathcal{C}_{\alpha}$ and $d_{\mathrm{TV}}(\mathcal{N}(\mu,\sigma^{2}),\mathcal{N}(\tilde{\mu},\tilde{\sigma}^{2}))\leq\frac{3}{2}\left(e^{\gamma}-1\right)+\frac{\beta}{4}\leq\alpha$, as required.

It only remains to show that the cover size is locally small. Let $\mu\in\mathbb{R}$ and $\sigma>0$.

[TABLE]

∎

Combining Lemma 6.14 with Lemma 6.9 and Theorem 4.1 yields the following.

Corollary 6.15**.**

Suppose we are given a set of samples $X_{1},\dots,X_{n}\sim P$, where $P$ is a univariate Gaussian distribution $\mathcal{N}(\mu,\sigma^{2})$. Then there exists a $(\varepsilon,\delta)$-differentially private algorithm which outputs a univariate Gaussian distribution $H^{*}$ such that $d_{\mathrm{TV}}(P,H^{*})\leq 7\alpha$ with probability $\geq 9/10$, so long as

[TABLE]

This sample complexity is comparable to to that of Karwa and Vadhan [KV18], who give an $(\varepsilon,\delta)$-DP algorithm with sample complexity $\tilde{O}\left(\frac{1}{\alpha^{2}}+\frac{1}{\alpha\varepsilon}+\frac{\log(1/\delta)}{\varepsilon}\right)$.

6.3 Sums of Independent Random Variables

In this section, we apply our results to distribution classes which are defined as the sum of independent (but not necessarily identical) distributions. These are all generalizations of the classical Binomial distribution, and they have enjoyed a great deal of study into the construction of sparse covers. To the best of our knowledge, we are the first to provide private learning algorithms for these classes.

We start with the Poisson Binomial distribution.

Definition 6.16**.**

A $k$-Poisson Binomial Distribution ($k$-PBD) is the sum of $k$ independent Bernoulli random variables.

We next consider sums of independent integer random variables, which generalize PBDs (which correspond to the case $d=2$).

Definition 6.17**.**

A $(k,d)$-Sum of Independent Integer Random Variables ($(k,d)$-SIIRV) is the sum of $k$ independent random variables over $\{0,\dots,d-1\}$.

Finally, we consider Poisson Multinomial distributions, which again generalize PBDs (which, again, correspond to the case $d=2$).

Definition 6.18**.**

A $(k,d)$-Poisson Multinomial Distribution ($(k,d)$-PMD) is the sum of $k$ independent $d$-dimensional categorical random variables, i.e., distributions over $\{e_{1},\dots,e_{d}\}$, where $e_{i}$ is the $i$th basis vector.

We start with a covering result for SIIRVs (including the special case of PBDs), which appears in [DKS16b]. Previous covers for PBDs and SIIRVs appear in [DP09, DP15b, DDO*+*13].

Lemma 6.19** ([DKS16b]).**

There exists an $\alpha$-cover of the set of $(k,d)$-SIIRVs of size

[TABLE]

Using this cover, we can apply Corollary 1.2 to attain the following learning result for PBDs and SIIRVs.

Corollary 6.20**.**

Suppose we are given a set of samples $X_{1},\dots,X_{n}\sim P$, where $P$ is $\alpha$-close to a $(k,d)$-SIIRV. Then for any constant $\zeta>0$, there exists an $\varepsilon$-differentially private algorithm which outputs a $(k,d)$-SIIRV $H^{*}$ such that $d_{\mathrm{TV}}(P,H^{*})\leq(6+2\zeta)\alpha$ with probability $\geq 9/10$, so long as

[TABLE]

Next, we move on to PMDs. The following cover does not appear verbatim in any single location, but is a combination of results from a few different sources. The proofs for the best bounds on first term appears in [DDKT16], the second in [DKT15], and the third in [DKS16a]. Larger covers previously appeared in [DP08, DP15a].

Lemma 6.21** ([DKT15, DDKT16, DKS16a]).**

For any $d>2$, there exists an $\alpha$-cover of the set of $(k,d)$-PMDs of size

[TABLE]

This implies the following learning result for PMDs.

Corollary 6.22**.**

Suppose we are given a set of samples $X_{1},\dots,X_{n}\sim P$, where $P$ is $\alpha$-close to a $(k,d)$-PMD, for any $d>2$. Then there exists an $\varepsilon$-differentially private algorithm which outputs a $(k,d)$-PMD $H^{*}$ such that $d_{\mathrm{TV}}(P,H^{*})\leq(6+2\zeta)\alpha$ with probability $\geq 9/10$, so long as

[TABLE]

6.4 Piecewise Polynomials

In this section, we apply our results to semi-agnostically learn piecewise polynomials. This class of distributions is very expressive, allowing us to approximate a wide range of natural distribution classes.

Definition 6.23**.**

A $(t,d,k)$-piecewise polynomial distribution is a distribution $P$ over $[k]$, such that there exists a partition of $[k]$ into $t$ disjoint intervals $I_{1},\dots,I_{t}$ such that on each interval $I_{j}\subseteq[k]$, the probability mass function of $P$ takes the form $p_{j}(x)=\sum_{i=0}^{d}c_{i}^{(j)}x^{i}$ for some coefficients $c_{i}^{(j)}$, for all $x\in I_{j}$.

We construct a cover for piecewise polynomials.

Lemma 6.24**.**

There exists a universal constant $c>0$ such that there is an $\alpha$-cover of the set of $(t,d,k)$-piecewise polynomials of size

[TABLE]

Proof.

We specify an element of the cover by

Selecting one of $\binom{k}{t-1}$ partitions of $[k]$ into $t$ intervals $I_{1},\dots,I_{t}$, and 2. 2.

For each interval $I_{j}$, selecting an element of an $(\alpha/t)$-cover $\mathcal{C}_{j}$ of the set of degree-$d$ polynomials over $I_{j}$ which are uniformly bounded by $1$.

The total size of the cover is $\binom{k}{t-1}\prod_{j=1}^{t}|\mathcal{C}_{j}|$. The theorem follows from Proposition 6.25 below, which constructs an $(\alpha/t)$-cover $\mathcal{C}_{j}$ of size at most $\left(\frac{tk\cdot e^{cd^{1/2}}}{\alpha}\right)^{d+1}$ for every interval $I_{j}$.

∎

Proposition 6.25**.**

There exist constants $b,c>0$ for which the following holds. Let $I\subseteq[k]$ be an interval and let $\mathcal{P}$ be the set of polynomials $p:I\to\mathbb{R}$ of degree $d$ such that $|p(x)|\leq 1$ for all $x\in I$. There exists an $\alpha$-cover of $\mathcal{P}$ of size

[TABLE]

The proof of Proposition 6.25 relies on two major results in approximation theory, which we now state.

Lemma 6.26** (Duffin and Schaeffer [DS41]).**

Let $p:[-1,1]\to\mathbb{R}$ be a polynomial such that $|p(x)|\leq 1$ for all $x$ of the form $x=\cos(j\pi/d)$ for $j=0,1,\dots,d$. Then $|p^{\prime}(x)|\leq d^{2}$ for all $x\in[-1,1]$.

Lemma 6.27** (Coppersmith and Rivlin [CR92]).**

There exist constants $a,b>0$ for which the following holds. Let $p:\mathbb{R}\to\mathbb{R}$ be a polynomial of degree $d$, and suppose that $|p(t)|\leq 1$ for all $t=0,1,\dots,m$. Then $|p(t)|\leq a\exp(bd^{2}/m)$ for all $t\in[0,m]$.

Proof of Proposition 6.25.

We consider two cases, corresponding to the two terms in the minimum. First, consider the function $f:I\to\mathbb{R}$ where $f(t)$ is obtained by rounding $p(t)$ to the nearest multiple of $\alpha/k$. Then $f$ satisfies $\sum_{t\in I}|f(t)-p(t)|\leq\alpha$. There are at most $(2k/\alpha)^{|I|}$ functions $f$ which can be constructed this way, giving the first term in the maximum.

For the second term, we construct a cover for $\mathcal{P}$ by approximately interpolating through $d+1$ carefully chosen points in the continuous interval corresponding to $I$. By applying an affine shift, we may assume that $I=\{0,1,\dots,m\}$ for some integer $m\leq k-1$. Let $p\in\mathcal{P}$ and for $x\in[0,m]$ let $\hat{p}(x)$ be the value of $p(x)$ rounded to the nearest integer multiple of $\alpha/(2kd^{2})$. Let $q:[0,m]\to\mathbb{R}$ be the unique degree-$d$ polynomial obtained by interpolating through the points $(x_{j},\hat{p}(x_{j}))$ where $x_{j}=m(1+\cos(j\pi/d))/2$ for $j=0,1,\dots,d$.

We first argue that the polynomial $q$ so defined satisfies $\sum_{t=0}^{m}|p(t)-q(t)|\leq\alpha$. Let $r(x)=p(x)-q(x)$ for $x\in[0,m]$. Then by construction, $|r(x_{j})|\leq\alpha/(2kd^{2})$ for all interpolation points $x_{j}$. By the Duffin-Schaeffer Inequality (Lemma 6.26), we therefore have $|r^{\prime}(x)|\leq\frac{\alpha}{km}$ for all $x\in[0,m]$. By the Fundamental Theorem of Calculus, $r(t)=r(0)+\int_{0}^{t}r^{\prime}(t)\ dt$ satisfies $|r(t)|\leq(t+1)\cdot\frac{\alpha}{km}\leq\alpha/k$, and hence $\sum_{t=0}^{m}|r(t)|\leq\alpha$.

We now argue that the set of polynomials $q$ that can be constructed in this fashion has size $(ckd^{2}\exp(bd^{2}/m)/\alpha)^{d+1}$. By the Coppersmith-Rivlin Inequality (Lemma 6.27), there are constants $a,b>0$ such that $|p(x)|\leq a\exp(bd^{2}/m)$ for all $x\in[0,m]$. Therefore, for each $p\in\mathcal{P}$ and each interpolation point $x_{j}$, there are at most $4a\cdot kd^{2}\exp(bd^{2}/m)/\alpha$ possible values that $\hat{p}(x_{j})$ can take. Hence, the polynomial $q$ can take one of at most $(4a\cdot kd^{2}\exp(bd^{2}/m)/\alpha)^{d+1}$ possible values, as we wanted to show. ∎

Lemma 6.28**.**

The VC dimension of $(t,d,k)$-piecewise polynomial distributions is at most $2t(d+1)$.

Proof.

Consider two piecewise polynomial distributions. The difference between their probability mass functions is a piecewise polynomial of degree $\leq d$. The number of intervals needed to represent this piecewise function is $\leq 2t$. It follows that this difference can change sign at most $2td+2t-1$ times – each polynomial can change sign at most $d$ times and the sign can change at the interval boundaries. Thus such a function cannot label $2td+2t+1$ points with alternating signs, which implies the VC bound. ∎

As a corollary, we obtain the following learning algorithm.

Corollary 6.29**.**

Suppose we are given a set of samples $X_{1},\dots,X_{n}\sim P$, where $P$ is $\alpha$-close to a $(t,d,k)$-piecewise polynomial. Then there exists an $\varepsilon$-differentially private algorithm which outputs a $(t,d,k)$-piecewise polynomial $H^{*}$ such that $d_{\mathrm{TV}}(P,H^{*})\leq(6+2\zeta)\alpha$ with probability $\geq 9/10$, so long as

[TABLE]

We compare with the work of Diakonikolas, Hardt, and Schmidt [DHS15]. They present an efficient algorithm for $(t,1,k)$-piecewise polynomials, with sample complexity $\tilde{O}\left(\frac{t}{\alpha^{2}}+\frac{t\log k}{\alpha\varepsilon}\right)$, which our algorithm matches777As stated in [DHS15], their algorithm guarantees approximate differential privacy, but swapping in an appropriate pure DP subroutine gives this result.. They also claim their results extend to $(t,d,k)$-piecewise polynomials, though no theorem statement is provided. While we have not investigated the details of this extension, we believe the resulting sample complexity should be qualitatively similar to ours, plausibly with the factor of $\frac{t(d+1)^{3/2}\log k}{\alpha\varepsilon}$ replaced by $\frac{t(d+1)\log k}{\alpha\varepsilon}$.

6.5 Mixtures

In this section, we show that our results immediately extend to learning mixtures of classes of distributions.

Definition 6.30**.**

Let $\mathcal{H}$ be some set of distributions. A $k$-mixture of $\mathcal{H}$ is a distribution with density $\sum_{i=1}^{k}w_{i}P_{i}$, where each $P_{i}\in\mathcal{H}$.

Our results follow roughly due to the fact that a cover for $k$-mixtures of a class can be written as the Cartesian product of $k$ covers for the class. More precisely, we state the following result which bounds the size of the cover of the set of $k$-mixtures.

Lemma 6.31**.**

Consider the class of $k$-mixtures of $\mathcal{H}$, where $\mathcal{H}$ is some set of distributions. There exists a $2\alpha$-cover of this class of size $\left|\mathcal{C}_{\alpha}\right|^{k}\left(\frac{k}{2\alpha}+1\right)^{k-1}$, where $\mathcal{C}_{\alpha}$ is an $\alpha$-cover of $\mathcal{H}$.

Proof.

Each element in the cover of the class of mixtures will be obtained by taking $k$ distributions from $\mathcal{C}_{\alpha}$, in combination with $k$ mixing weights, which are selected from the set $\left\{0,\frac{2\alpha}{k},\frac{4\alpha}{k},\dots,1\right\}$, such that the sum of the mixing weights is $1$. The size of this cover is $|\mathcal{C}_{\alpha}|^{k}\cdot\left(\frac{k}{2\alpha}+1\right)^{k-1}$. We reason about the accuracy of the cover as follows. Fix some mixture of $k$ distributions as $\sum_{i=1}^{k}w^{(1)}_{i}P^{(1)}_{i}$, and we will reason about the closest element in our cover, $\sum_{i=1}^{k}w^{(2)}_{i}P^{(2)}_{i}$. By triangle inequality, we have that

[TABLE]

Since $\mathcal{C}_{\alpha}$ is an $\alpha$-cover and $\sum_{i=1}^{k}w^{(1)}_{i}=1$, the total variation distance incurred by the second term will be at most $\alpha$. As for the mixing weights, note that for the first $k-1$ weights, the nearest weight is at distance at most $\frac{\alpha}{k}$, contributing a total of less than $\frac{\alpha}{2}$. The last mixing weight can be rewritten in terms of the sum of the errors of the other mixing weights, similarly contributing another total of less than $\frac{\alpha}{2}$. This results in the total error being at most $2\alpha$, as desired. ∎

With this in hand, the following corollary is almost immediate from Corollary 1.2. The factor of $(9+3\zeta)\alpha$ (as opposed to $(6+2\zeta)\alpha$) is because the closest distribution in the cover of mixture distributions is $3\alpha$-close to be $P$ (rather than $2\alpha$).

Corollary 6.32**.**

Let $X_{1},\dots,X_{n}\sim P$, where $P$ is $\alpha$-close to a $k$-mixture of distributions from some set $\mathcal{H}$. Let $\mathcal{C}_{\alpha}$ be an $\alpha$-cover of the set $\mathcal{H}$, and $\zeta>0$ be a constant. There exists an $\varepsilon$-differentially private algorithm which outputs a distribution which is $(9+3\zeta)\alpha$-close to $P$ with probability $\geq 9/10$, as long as

[TABLE]

For example, instantiating this for mixtures of Gaussians (and disregarding terms which depend on $R$ and $\kappa$), we get an algorithm with sample complexity $\tilde{O}\left(\frac{kd^{2}}{\alpha^{2}}+\frac{kd^{2}}{\alpha\varepsilon}\right)$.

6.6 Supervised Learning

We describe an application of our results to the task of binary classification, as modeled by differentially private PAC learning [KLN*+*11]. Let $\mathcal{F}=\{f:X\to\{0,1\}\}$ be a publicly known concept class of Boolean functions over a domain $X$. Let $P$ be an unknown probability distribution over $X$, and let $f$ be an unknown function from $\mathcal{F}$. Given a sequence $\{(x_{i},f(x_{i}))\}_{i=1}^{n}$ of i.i.d. samples from $P$ together with their labels under $f$, the goal of a PAC learner $L$ is to identify a hypothesis $h:X\to\{0,1\}$ such that $\Pr_{x\sim P}[h(x)\neq f(x)]\leq\alpha$ for some error parameter $\alpha>0$. We say that $L$ is $(\alpha,\beta)$-accurate if for every $f\in\mathcal{F}$ and every distribution $P$, it is able to identify such a hypothesis $h$ with probability at least $1-\beta$ over the choice of the sample and any internal randomness of $L$.

One of the core results of statistical learning theory is that the sample complexity of non-private PAC learning is characterized, up to constant factors, by the VC dimension of the concept class $\mathcal{F}$. When one additionally requires the learner $L$ to be differentially private with respect to its input sample, such a characterization is unknown. However, it is known that the sample complexity of private learning can be arbitrarily higher than that of non-private learning. For example, when $\mathcal{F}=\{f_{t}:t\in X\}$ is the class of threshold functions defined by $f_{t}(x)=1\iff x\leq t$ over a totally ordered domain $X$, the sample complexity of PAC learning under the most permissive notion of $(\varepsilon,\delta)$-differential privacy is $\Omega(\log^{*}|X|)$ [BNSV15, ALMM19]. Meanwhile, the VC dimension of this class, and hence the sample complexity of non-private learning, is a constant independent of $|X|$.

While this separation shows that there can be a sample cost of privacy for PAC learning, this cost can be completely eliminated if the distribution $P$ on examples is known. This was observed by Beimel, Nissim, and Stemmer [BNS16], who showed that if a good approximation to $P$ is known, e.g., from public unlabeled examples or from differentially private processing of unlabeled examples, then the number of labeled examples needed for private PAC learning is only $O(VC(\mathcal{F}))$.

Theorem 6.33**.**

Let $\varepsilon>0$, $\mathcal{F}=\{f:X\to\{0,1\}\}$, and $P$ be a publicly known distribution over $X$. For $n=O\left(\frac{1}{\alpha^{2}\varepsilon}(VC(\mathcal{F})\log(1/\alpha)+\log(1/\beta))\right)$, there exists an $\varepsilon$-differentially private algorithm $L:(X\times\{0,1\})^{n}\to\mathcal{F}$ such that for every $f\in\mathcal{F}$, with probability at least $1-\beta$ over the choice of $x_{1},\dots,x_{n}\leftarrow P$, we have that $L((x_{1},f(x_{1})),\dots,(x_{n},f(x_{n})))$ produces $h\in\mathcal{F}$ such that $\Pr_{x\sim P}[f(x)\neq h(x)]\leq\alpha$.

Our results suggest a natural two-step algorithm for private PAC learning when the distribution $P$ itself is not known, but is known to (approximately) come from a set of distributions $\mathcal{H}$: The algorithm first uses private hypothesis selection to select $\hat{H}$ with $d_{\mathrm{TV}}(P,\hat{H})\leq\alpha/2$, and then runs the algorithm of [BNS16] using $\hat{H}$ in place of $P$ with error parameter $\alpha/2$. Using the fact that $d_{\mathrm{TV}}(P,\hat{H})\leq\alpha/2$ implies $|\Pr_{x\sim P}[f(x)\neq h(x)]-\Pr_{x\sim\hat{H}}[f(x)\neq h(x)]|\leq\alpha/2$, the following result holds by combining Theorem 6.33 with Corollary 1.2.

Corollary 6.34**.**

Let $\mathcal{H}$ be a set of distributions over $X$ with an $\alpha$-cover $\mathcal{C}_{\alpha}$. Let $P$ be a distribution over $X$ with $d_{\mathrm{TV}}(P,\mathcal{H})\leq\alpha/(4(3+\zeta))$. Then for

[TABLE]

there exists an $\varepsilon$-differentially private algorithm $L:(X\times\{0,1\})^{n}\to\mathcal{F}$ such that for every $f\in\mathcal{F}$, with probability at least $3/4$ over the choice of $x_{1},\dots,x_{n}\leftarrow P$, we have that $L((x_{1},f(x_{1})),\dots,(x_{n},f(x_{n})))$ produces $h\in\mathcal{F}$ such that $\Pr_{x\sim P}[f(x)\neq h(x)]\leq\alpha$.

Theorem 6.33 can, of course, also be combined with the more refined guarantees of Theorem 4.1. As an example application, combining Theorem 6.33 with Corollary 6.13 gives a $(\varepsilon,\delta)$-differentially private algorithm for learning one-dimension thresholds with respect to univariate Gaussian distributions on the reals. In contrast, this task is impossible without making distributional assumptions.

7 Conclusions

In this paper, we presented differentially private methods for hypothesis selection. The sample complexity can be bounded by the logarithm of the number of hypotheses. This allows us to provide bounds on the sample complexity of (semi-agnostically) learning a class which depend on the logarithm of the covering number, complementing known lower bounds which depend on the logarithm of the packing number. There are many interesting questions left open by our work, a few of which we outline below.

Our algorithms for learning classes of distributions all use cover-based arguments, and thus are not computationally efficient. For instance, we provide the first $\tilde{O}(d)$ sample complexity upper bound on $\varepsilon$-differentially privately learning a product distribution and Gaussian with known covariance. One interesting question is whether there is an efficient algorithm which achieves this sample complexity. 2. 2.

The running time of our method is quadratic in the number of hypotheses – is it possible to reduce this to a near-linear time complexity? 3. 3.

Our main theorem obtains an approximation factor which is arbitrarily close to $3$, which is optimal for this problem, even without privacy. This factor can be reduced to 2 if one is OK with outputting a mixture of hypotheses from the set [BKM19]. Is this achievable with privacy constraints?

Acknowledgments

The authors would like to thank Shay Moran for bringing to their attention the application to PAC learning mentioned in Section 6.6, Jonathan Ullman for asking questions which motivated Remark 6.4, and Clément Canonne for assistance in reducing the constant factor in the approximation guarantee.