Differentially Private Nonparametric Hypothesis Testing

TL;DR

This paper develops differentially private versions of classical nonparametric hypothesis tests, improving statistical power while maintaining privacy, applicable to various data types without distributional assumptions.

Contribution

Introduces novel differentially private nonparametric tests, including private analogues of Kruskal-Wallis, Mann-Whitney, Wilcoxon, and t-tests, with significant power improvements.

Findings

Private tests outperform prior methods in statistical power.

Nonparametric tests maintain effectiveness without distributional assumptions.

Results show large power gains even when parametric assumptions hold.

Abstract

Hypothesis tests are a crucial statistical tool for data mining and are the workhorse of scientific research in many fields. Here we study differentially private tests of independence between a categorical and a continuous variable. We take as our starting point traditional nonparametric tests, which require no distributional assumption (e.g., normality) about the data distribution. We present private analogues of the Kruskal-Wallis, Mann-Whitney, and Wilcoxon signed-rank tests, as well as the parametric one-sample t-test. These tests use novel test statistics developed specifically for the private setting. We compare our tests to prior work, both on parametric and nonparametric tests. We find that in all cases our new nonparametric tests achieve large improvements in statistical power, even when the assumptions of parametric tests are met.

Tables

Table 1. Table 1: Critical Value Comparison for n = 100 𝑛 100 n=100

$ϵ$	$\alpha$	Public	New	TC
1	0.1	1.282	1.417	2.680
	0.05	1.645	1.826	3.091
	0.025	1.960	2.186	3.511
0.1	0.1	1.282	5.684	14.786
	0.05	1.645	8.063	15.197
	0.025	1.960	10.438	15.617
0.01	0.1	1.282	55.350	135.843
	0.05	1.645	79.233	136.254
	0.025	1.960	103.116	136.674

Table 2. Table 2: Critical Value Comparison for n = 1000 𝑛 1000 n=1000

$ϵ$	$\alpha$	Public	New	TC
1	0.1	1.282	1.296	1.763
	0.05	1.645	1.665	2.174
	0.025	1.960	1.984	2.594
0.1	0.1	1.282	2.203	5.617
	0.05	1.645	2.975	6.028
	0.025	1.960	3.740	6.448
0.01	0.1	1.282	17.681	44.157
	0.05	1.645	25.234	44.568
	0.025	1.960	32.844	44.988

Table 3. Table 3: New Critical Value Table for ϵ italic-ϵ \epsilon = 1.0 1.0 1.0

$n$	0.05	0.025	0.01	0.005
10	70	83	102	116
20	155	183	220	248
30	256	299	355	397
40	369	429	506	562
50	494	572	670	742
75	854	984	1143	1257
100	1271	1460	1690	1853
200	3402	3895	4486	4900
300	6127	7012	8069	8798
400	9335	10679	12276	13382
500	12978	14845	17061	18592
1000	36235	41443	47637	51906

Table 4. Table 4: New Critical Value Table for ϵ italic-ϵ \epsilon = 0.1 0.1 0.1

$n$	0.05	0.025	0.01	0.005
10	600	739	922	1061
20	1202	1479	1846	2123
30	1806	2220	2770	3185
40	2413	2968	3704	4261
50	3018	3713	4628	5324
75	4541	5577	6954	7989
100	6073	7461	9294	10677
200	12328	15098	18767	21531
300	18733	22892	28391	32519
400	25296	30837	38193	43736
500	32054	38979	48128	55083
1000	68258	82120	100408	114230

Table 5. Table 5: New Critical Value Table for ϵ italic-ϵ \epsilon = 0.01 0.01 0.01

$n$	0.05	0.025	0.01	0.005
10	5992	7377	9209	10596
20	11971	14742	18416	21196
30	17976	22137	27644	31774
40	23974	29516	36877	42425
50	29964	36905	46081	53034
75	44933	55371	69105	79513
100	59921	73792	92066	106005
200	119902	147619	184222	212010
300	179942	221477	276678	317895
400	239695	295106	368374	423528
500	299627	368763	460256	529522
1000	600096	738071	921529	1061150

Equations

$$\mathrm{Pr}[T\ge t\mid T=f(\text{X})\text{ and }\text{X}\leftarrow H_0]=p.$$

$$\mathrm{Pr}[T\ge t^{\ast }\mid T=f(\text{X})\text{ and }\text{X}\leftarrow H_A].$$

$$\mathrm{Pr}[\tilde{f}(\text{x})\in \mathcal{S}]\le e^{ϵ}\mathrm{Pr}[\tilde{f}({\text{x}}^{\prime })\in \mathcal{S}]+\delta .$$

$$\textsf{Lap}(x|b)=\frac{1}{2b}\textup{exp}\Big{(}-\frac{|x|}{b}\Big{)}.$$

$$G{S}_f=\underset{\text{x},{\text{x}}^{\prime }}{\max }|f(\text{x})-f({\text{x}}^{\prime })|,$$

$$\tilde{f}(\text{x})=f(\text{x})+Y,$$

$$L{S}_f(\text{x})=\underset{{\text{x}}^{\prime }}{\max }|f(\text{x})-f({\text{x}}^{\prime })|,$$

$$h=(n-1)\frac{{\sum }_{i=1}^g{n}_i({\bar{r}}_i-\bar{r}{)}^2}{{\sum }_{i=1}^g{\sum }_{j=1}^{n_i}(r_{ij}-\bar{r}{)}^2}.$$

$$h=\frac{12}{n(n+1)}\sum _{i=1}^g{n}_i{\bar{r}}_i^2-3(n+1).$$

$$h_{abs}=(n-1)\frac{{\sum }_{i=1}^g{n}_i|{\bar{r}}_i-\bar{r}|}{{\sum }_{i=1}^g{\sum }_{j=1}^{n_i}|r_{ij}-\bar{r}|}.$$

$$h_{abs}=\{\begin{array}{ll}\frac{4(n-1)}{n^2}\sum _{i=1}^g{n}_i\left|{\bar{r}}_i-\frac{n+1}{2}\right|,& \text{mbox}ifn\text{mbox}iseven\\ \frac{4}{n+1}\sum _{i=1}^g{n}_i\left|{\bar{r}}_i-\frac{n+1}{2}\right|,& \text{mbox}ifn\text{mbox}isodd\end{array}$$

$$\mathrm{Pr}[\tilde{W}>b+g]$$

$$=\mathrm{Pr}[W>b]+\mathrm{Pr}[\Lambda >g]$$

$$-\mathrm{Pr}[W>b\text{ and }\Lambda >g]$$

$$=\beta +\gamma -\beta \gamma$$

$$\sum _{k=a}^{b}k=\frac{(b-a+1)(b+a)}{2}$$

$$\frac{n_i(n_i+1)}{2}\le \sum _{j=1}^{n_i}{r}_{ij}\le \frac{n_i(2n-n_i+1)}{2}$$

$$\sum _{j=1}^{n_i}{r}_{ij}$$

$$=\frac{(n-(n-n_i+1)+1)(n+(n-n_i+1))}{2}$$

$$=\frac{n_i(2n-n_i+1)}{2}$$

$$\sum _{j=1}^{n_i}{r}_{ij}$$

$$=\frac{(n_i-1+1)(n_i+1)}{2}$$

$$=\frac{n_i(n_i+1)}{2}$$

$$\frac{1+n_i}{2}\le {\bar{r}}_i\le \frac{2n-n_i+1}{2},$$

$$|h^{\prime }-h|$$

$$\displaystyle=\frac{12}{n(n+1)}\bigg{|}\sum_{i=1}^{g}n_{i}\big{(}\bar{r^{\prime}}_{i}^{2}-\bar{r}_{i}^{2}\big{)}\bigg{|}$$

$$\displaystyle=\frac{12}{n(n+1)}\bigg{|}\sum_{i=1}^{g}n_{i}\bigg{(}\frac{1}{n_{i}^{2}}\bigg{(}\sum_{j=1}^{n_{i}}r^{\prime}_{ij}\bigg{)}^{2}-\frac{1}{n_{i}^{2}}\bigg{(}\sum_{j=1}^{n_{i}}r_{ij}\bigg{)}^{2}\bigg{)}\bigg{|}$$

$$\displaystyle=\frac{12}{n(n+1)}\bigg{|}\sum_{i=1}^{g}\frac{1}{n_{i}}\bigg{(}\bigg{(}\sum_{j=1}^{n_{i}}r^{\prime}_{ij}\bigg{)}^{2}-\bigg{(}\sum_{j=1}^{n_{i}}r_{ij}\bigg{)}^{2}\bigg{)}\bigg{|}$$

$$\displaystyle\leq\frac{12}{n(n+1)}\sum_{i=1}^{g}\bigg{|}\frac{1}{n_{i}}\bigg{(}\bigg{(}\sum_{j=1}^{n_{i}}r^{\prime}_{ij}\bigg{)}^{2}-\bigg{(}\sum_{j=1}^{n_{i}}r_{ij}\bigg{)}^{2}\bigg{)}\bigg{|}$$

$$z_{i}=\frac{1}{n_{i}}\bigg{|}\bigg{(}\sum_{j=1}^{n_{i}}r^{\prime}_{ij}\bigg{)}^{2}-\bigg{(}\sum_{j=1}^{n_{i}}r_{ij}\bigg{)}^{2}\bigg{|}$$

$$z_1$$

$$\displaystyle=\frac{1}{n_{1}}\Big{|}\pm 2(d-a_{1})\sum_{j=1}^{n_{i}}r_{ij}+(d-a_{1})^{2}\Big{|}$$

$$\displaystyle=\frac{1}{n_{1}}\Big{|}\pm 2(d-a_{1})n_{1}\bar{r}_{1}+(d-a_{1})^{2}\Big{|}$$

$$\displaystyle=\Big{|}\frac{(d-a_{1})^{2}}{n_{1}}\pm 2\cdot(d-a_{1})\cdot\bar{r}_{1}\Big{|}$$

$$\le \frac{(n-1{)}^2}{n_1}+2\cdot (n-1)\cdot {\bar{r}}_1$$

$$\le (n-1{)}^2+2\cdot (n-1)\cdot \frac{2n-n_1+1}{2}$$

$$\le (n-1{)}^2+2\cdot (n-1)\cdot n$$

$$\sum _{i=2}^g|z_i|$$

$$\displaystyle=\sum_{i=2}^{g}\frac{1}{n_{i}}\Big{|}a_{i}^{2}\pm 2a_{i}\sum_{j=1}^{n_{i}}r_{ij}\Big{|}$$

Full text

Differentially Private Nonparametric Hypothesis Testing

Simon Couch

Zeki Kazan

Kaiyan Shi

couchs@reed.edu

kazanz@reed.edu

kaishi@reed.edu

Andrew Bray

Adam Groce

abray@reed.edu

agroce@reed.edu

Reed College Mathematics Department

Portland, OR, USA

Abstract

Hypothesis tests are a crucial statistical tool for data mining and are the workhorse of scientific research in many fields. Here we study differentially private tests of independence between a categorical and a continuous variable. We take as our starting point traditional nonparametric tests, which require no distributional assumption (e.g., normality) about the data distribution. We present private analogues of the Kruskal-Wallis, Mann-Whitney, and Wilcoxon signed-rank tests, as well as the parametric one-sample t-test. These tests use novel test statistics developed specifically for the private setting. We compare our tests to prior work, both on parametric and nonparametric tests. We find that in all cases our new nonparametric tests achieve large improvements in statistical power, even when the assumptions of parametric tests are met.

Introduction

In 2011, researchers in Switzerland began an investigation into the link between methylation levels of a given gene and the occurance of schizophrenia and bipolar disorder[6]. They recruited patients that suffered from psychosis as well as healthy controls and measured the level of methylation of the gene in each individual. They found that levels in the groups suffering from psychosis were higher than those in the healthy group. In order to rule out the possibility that this difference was due to sampling variability, they relied upon a suite of nonparameteric hypothesis tests to establish that this link likely exists in the general population.

While the results of these tests were published in an academic journal, the data itself is unavailable to preserve the privacy of the patients. This is a necessary consideration when working with sensitive data, but it hampers scientific reproducibility and the extension of this work by other researchers.

Our goal in this paper is to provide nonparametric hypothesis tests that satisfy differential privacy. The difficulty of developing a private test comes not just from the need to privately approximate a test statistic, but also from the need for an accurate reference distribution that will produce valid p-values. It is not sufficient to treat the approximate test statistic as equivalent to its non-private counterpart.

In this paper, we present several new hypothesis tests. In all cases we are considering data sets with a categorical explanatory variable (e.g., membership in the schizophrenic, bipolar, or control group) and a continuous dependent variable (e.g., methylation level). Our goal when designing a hypothesis test is to maximize the statistical power of the test, or equivalently to minimize the amount of data needed to detect a particular effect.

In the traditional public setting, there are two families of tests for these scenarios. The more commonly used are parametric tests that assume that within each group, the continuous variable follows a particular distribution (usually Gaussian). An alternative to these tests are nonparametric tests, which make no distributional assumption but, in exchange, have slightly lower power. Nonparametric tests generally rely upon substituting in, for each data point, the rank of the continuous variable relative to the rest of the sample. The test statistic is then a function of these ranks rather than the original values.

The private hypothesis tests we propose all use rank-based test statistics. Our overarching argument in this paper, beyond the individual value of each of the tests we introduce, is that in the private setting these rank-based test statistics are more powerful than the traditional parametric alternatives. This is contrary to the public setting, where the parametric tests (when their assumptions are met) perform better.

Our second, broader point is that as a research community we need to support the development of hypothesis tests specifically tailored to the private setting. Our private test statistics are not simply approximations of traditional test statistics from the public setting and as a result, we find that they can require an order of magnitude less data. Current tests used by statisticians in the public setting have been refined through decades of incremental improvement, and the same sort of development needs to happen in the private setting.

Our contributions

We introduce several new private hypothesis tests that mirror the three most commonly used rank-based tests. In one setting, this is a private analog of the traditional public test statistic, but for the remaining two settings it is a new statistic developed specifically for the private setting. The privacy of these statistics generally follows from non-trivial but reasonably straightforward applications of the Laplace mechanism. Our main contribution is not the method of achieving privacy but the construction of novel private hypothesis tests with high statistical power.

There are two components to the construction of each hypothesis test. The first is the creation of a test statistic to capture the effect of interest while remaining provably private. The second is a method to learn the distribution of the statistic under the null hypothesis in order to compute p-values, which are the object of primary interest to researchers.

In particular, we develop tests for the following cases:

Three or more groups

When the categorical variable divides the sample into three or more groups, the traditional public test is the one-way analysis of variance (ANOVA) in the parametric case and Kruskal-Wallace in the nonparametric case. ANOVA has been previously studied by Campbell et al. [5] and then by Swanberg et al. [31], who improved the power by an order of magnitude. We give the first private nonparametric test by modifying the rank-based Kruskal-Wallace test statistic for the private setting. We provide experimental evidence that this statistic has dramatically higher power than a simple privatized version of the public Kruskal-Wallace statistic. Moreover, we provide evidence that even in the parametric setting (i.e., when the data is normally distributed) the private rank-based statistic outperforms the private ANOVA test, in one representative case requiring only 23% as much data to reach the same power.

Two groups

In the two group setting, the most common public nonparametric test is the Mann-Whitney test. We provide an algorithm to release an approximation of this statistic under differential privacy and a second algorithm to conduct the test and release a valid p-value.

In the public setting, the analogous parametric test is the two sample $t$-test which is equivalent to an ANOVA test when there are two groups. We compare therefore to the private ANOVA test and show experimental evidence that our Mann-Whitney analog has significantly higher power.

Paired data

We also consider the case where the two sets of data are in correspondence with each other (e.g., before-and-after measurements). The nonparametric test in this case is the Wilcoxon signed-rank test, and it is the only nonparametric test that has previously been studied in the private setting[33]. We provide two improvements to the prior work. First, we change the underlying statistic to the less well-known Pratt variant, which we find conforms more easily to the addition of noise. Second, we show that our simulation method for computing reference distributions is more precise than the upper bounds used in the prior work (which contained an error which we identify and correct). The result is a significant improvement in the power of the test over the the prior work.

In parallel with the previous two scenarios, we then compare our private nonparametric test with a private version of the analogous parametric test, the paired $t$-test. A direct private implementation of this test does not exist in the literature, so we propose one here.111In simultaneous work Gaboardi et al. [15] propose such a test, but our test is still higher power. See Section 5.5 for more details. In alignment with the previous results, we show experimental evidence that the rank-based test has superior power.

For all our tests we give not just private test statistics but also precise methods of computing a reference distribution and a p-value, the final output practitioners actually need. We also experimentally verify that the probability of Type 1 errors (incorrectly rejecting the null hypothesis) is acceptably low. We give careful power analyses and use these to compare tests to each other. All our tests and experiments are implemented with publicly available code.222Our source code is available at: github.com/simonpcouch/non-pm-dpht

We find that rank-based statistics are very amenable to the private setting. We also repeatedly find that what is optimal in the public setting is no longer optimal in the private setting. We hope that this work contributes to the development of a standard set of powerful hypothesis tests that can be used by scientists to enable inferential analysis while protecting privacy.

Background

In this section, we begin by discussing hypothesis testing in general and outlining the formalities of differential privacy. We then discuss the difficulties of hypothesis testing within the privacy framework and the previous work done in this area. Each of our main results requires a more detailed discussion of prior work on that particular test or use case; we leave those discussions for Sections 3-5.

Hypothesis Testing

The key inferential leap that is made in hypothesis testing is the claim that not only is the sample of data incompatible with a particular scientific theory, but that it is evidence that this incompatible holds in a broader population. In the study on psychotic disease, the researchers used this technique to generalize from their 165 subjects to the population of causasian-descended Swiss. The scientific theory that they refuted, that there is no link between methylation levels and psychosis, is called the null hypothesis ($H_{0}$).

To test whether or not the data is consistent with $H_{0}$, a researcher computes a test statistic. The choice of a function $f$ to compute the test statistic largely determines the hypothesis test being used. For a random database X drawn according to $H_{0}$, the distribution of the statistic $T=f(\textbf{X})$ can be determined either analytically or through simulation. The researcher then computes a p-value, the probability that the observed test statistic or more extreme would occur under $H_{0}$.

Definition 2.1**.**

For a given test statistic $t=f(\textbf{x})$ and null hypothesis $H_{0}$, the p-value is defined as

[TABLE]

If the function $f$ is well-chosen, then the more the underlying distribution of X differs from the distribution under $H_{0}$, the more likely a low p-value becomes. Typically a threshold value $\alpha$ is chosen, such that we reject $H_{0}$ as a plausible explanation of the data when $p<\alpha$. The choice of $\alpha$ determines the type I error rate, the probability of incorrectly rejecting a true null hypothesis.

We define the critical value $t^{*}$ to be the value of the test statistic $t$ where $p=\alpha$. We use this to define the statistical power, a measure of how likely a hypothesis test is to pick up on a given effect (i.e. the chance of rejecting when the null hypothesis is false). The power is a function of how much the underlying distribution of X differs from the distribution under $H_{0}$ as well as the size of the database.

Definition 2.2**.**

For a given alternate data distribution $H_{A}$, the statistical power of a hypothesis test is

[TABLE]

Statistical power is the accepted metric by which the statistics community judges the usefulness of a hypothesis test. It provides a common scale upon which to evaluate different tests for the same use case.

Differential Privacy

To persuade people to allow their personal data to be collected, data owners must protect information about specific individuals. Historically, ad-hoc database anonymization techniques have been used (i.e. changing names to numeric IDs, rounding geospatial coordinates to the nearest block, etc.), but these methods have repeatedly been shown to be ineffective [17, 22, 32].

Differential privacy, proposed by Dwork et al. in 2006 [11], is a mathematically robust definition of privacy preservation, which guarantees that a query does not reveal anything about an individual as a consequence of their presence in the database. When the condition of differential privacy is satisfied, there is not much difference between the output obtained from the original database, and that obtained from a database that differs by only one individual’s data. Here we present $(\epsilon,\delta)$-differential privacy [10], which allows the closeness of output distributions to be measured with both a multiplicative and an additive factor. However, for most of our hypothesis tests $\delta=0$.

Definition 2.3** (Differential Privacy).**

A randomized algorithm $\tilde{f}$ on databases is $(\epsilon,\delta)$-differentially private if for all $\mathcal{S}\subseteq\textup{Range}(\tilde{f})$ and for databases $\textbf{x},\textbf{x}^{\prime}$ that only differ in one row:

[TABLE]

We call databases x and $\textbf{x}^{\prime}$ neighboring if they differ only in that a single row is altered (but not added or deleted), and we will use this notation in the following sections.333This is one of two roughly equivalent variants of differential privacy. The key difference is that under this definition the size of the database is public knowledge.

Differential privacy, like any acceptable privacy definition, is resistant to post processing. That is, if an algorithm is differentially private, an adversary with no access to the database will be unable to violate such privacy through further analysis (e.g., attempted deanonymization) of the query output [11].

Theorem 2.4** (Post Processing).**

Let $\tilde{f}$ be an $(\epsilon,\delta)$-differentially private randomized algorithm. Let $g$ be an arbitrary randomized algorithm. Then $g\circ\tilde{f}$ is $(\epsilon,\delta)$- differentially private.

Theorem 2.4 has another useful consequence. It allows us to develop private algorithms by first computing some private output, and then carrying out further computation on that output without accessing the database. The additional computation need not be analyzed carefully—the final output of the additional analysis is automatically known to retain privacy.

Differential privacy requires the introduction of some randomness to any query output. A frequently used method is the Laplace mechanism, introduced by Dwork et al. [11]. When given an arbitrary algorithm $f$ with real-valued output, this mechanism will add some noise drawn from the Laplace distribution to the output of the algorithm and release a noisy output.

Definition 2.5** (Laplace Distribution).**

The Laplace Distribution centered at 0 with scale $b$ is the distribution with probability density function:

[TABLE]

We write $\textsf{Lap}(b)$ to denote the Laplace distribution with scale $b$.

The scale of the Laplace Distribution used to produce the noisy output depends on the global sensitivity of the given algorithm $f$, which is the maximum change on the output of $f$ that could result from the alteration of a single row.

Definition 2.6** (Global sensitivity).**

The global sensitivity of a function $f$ is:

[TABLE]

where x and $\textbf{x}^{\prime}$ are neighbouring databases.

With computed sensitivity $GS_{f}$ and privacy parameter $\epsilon$, the Laplace mechanism applied to $f$ ensures $(\epsilon,0)$-differential privacy [11].

Definition 2.7** (Laplace Mechanism).**

Given any function $f$, the Laplace mechanism is defined as:

[TABLE]

where $Y$ is drawn from $\textsf{Lap}(GS_{f}/\epsilon)$, and $GS_{f}$ is the global sensitivity of $f$.

Theorem 2.8**.**

(Laplace Mechanism) The Laplace mechanism preserves $(\epsilon,0)$ differential privacy.

Global sensitivity is the maximum effect that can be caused by changing a single row of any database. Sometimes it is helpful to talk about local sensitivity for a given database x [24]. This is the maximum effect that can be caused by changing a row of that particular database.

Definition 2.9** (Local Sensitivity).**

The sensitivity of a function $f$ at a particular database x is:

[TABLE]

where $\textbf{x}^{\prime}$ is a neighboring database.

Note that $GS_{f}=\max_{\textbf{x}}LS_{f}(\textbf{x})$. Local sensitivity cannot simply be used in the Laplace mechanism in place of global sensitivity, because local sensitivity itself is a function of the database and therefore cannot be released. But private upper bounds on local sensitivity can be used to create similar mechanisms that do preserve privacy, and one of our algorithms uses just such a technique.

Choosing $\epsilon$ is an important consideration when using differential privacy. We consider several values of $\epsilon$ throughout our power analyses. The lowest, .01 is an extremely conservative privacy parameter and allows for safe composition with many other queries of comparable $\epsilon$ value. We also use $\epsilon$s of $.1$ and $1$, which, while higher, still provide very meaningful privacy protection. Ultimately, the choice of $\epsilon$ is a question of policy and depends on the relative importance with which privacy and utility are regarded. We also measure, for comparison, the power of the public versions of each test (equivalent to an $\epsilon$ of $\infty$). As one might expect, the amount of data needed to detect a given effect often scales roughly with the inverse of $\epsilon$.

Differentially Private Hypothesis Testing

Performing hypothesis tests within the framework of differential privacy introduces new complexity. A function to compute a private test statistic (be it a private version of a standard test statistic or an entirely new test statistic) is not useful on its own. We need a p-value or other understandable output, and that means understanding the reference distribution (i.e., the distribution of the statistic given $H_{0}$).

In classical statistics, test statistics are computed with deterministic functions. The randomness added to the test statistic in order to privatize it introduces new complexity. Most importantly, it causes the reference distribution to change. One cannot simply compare the private test statistic to the usual reference distribution, as the addition of noise can inflate the type I error well above acceptable levels [5].

Because of this, a complete differentially private hypothesis test requires not only a function for computing a private test statistic, but also a method for determining its null distribution. Often the exact reference distribution cannot be determined, so worst-case reference distributions or upper bounds on the resulting critical value must be used, and the precision of this reference distribution can have a large effect on the resulting power.

The goal of differentially private hypothesis test design is to develop a test with power as close as possible to the public test.

Related Work

There is a substantial and growing literature on differentially private hypothesis testing. One area of research is the study of the rate of convergence of private statistics to the distributions of their public analogues [28, 29, 37]. These papers do not offer practical, implementable tests and discussion of reference distributions when the noise is not yet negligible is often limited or entirely absent. Further, the results are often entirely asymptotic, without regard for constants that may prove to be problematic.

The chi-squared test, which tests the independence of two categorical variables,444This is the chi-squared test of independence. There are several related tests that use the same statistic, the chi-squared. has been the subject of much study, resulting in the development of many private variants. One of these works, that of Vu and Slavkovic [35], provides methods for calculation of accurate p-values adjusted for the addition of Laplace noise for differentially private single proportion and chi-squared tests specifically for clinical trial data. Several other papers, though they make asymptotic arguments on the uniformity of their p-values, have developed frameworks for private chi-squared tests specifically for the intent of genome-wide association study (GWAS) data [13, 18, 34]. For these same tests, Monte Carlo simulation has been shown to offer more precise analysis in some cases [14, 36]. There has also been work, like that of Rogers and Kifer [26], that proposes entirely new test statistics with asymptotic distributions more similar to their public counterparts.

While the development of private test statistics has achieved much attention, careful evaluations of statistical power of these new test statistics is not always demonstrated. This is unfortunate, as the cost of privacy (utility loss) must be accurately quantified in order for the widespread adoption or implementation of any of these methods. Fortunately, rigorous power analysis seems to be more common in recent work. Awan and Slavkovic recently presented a test for simple binomial data [2]. While the setting is the simplest possible, their paper gives what we believe is the first private test to come with a proof of optimality, something normally very difficult to achieve even in the public setting.

The body of work on numerical (rather than categorical) methods is less extensive but has been growing quickly in recent years. In 2017, Nguyen and Hui proposed algorithms for survival analysis methods [23]. There have been frameworks developed for testing the difference in means of normal distributions [8, 9], and for testing whether a sample is consistent with a normal distribution with a particular mean [30]. Differentially private versions of linear regressions, a class of inference that is extremely common in many fields both within and outside of academia, have received a notable level of attention, but the treatment of regression coefficients as test statistics has come about only recently [3, 27]. Two works have studied differentially private versions of one-way analysis of variance (ANOVA) [5, 31]. The only prior work done on nonparametric hypothesis tests, as far as we are aware, is on the Wilcoxon signed-rank test by Task and Clifton in 2016 [33]. Prior work specifically relevant to the tests we are proposing will be discussed in more detail in the relevant section.

Many groups

We first consider the most general case, where we wish to distinguish whether many groups share the same distribution on a continuous variable. The standard parametric test in the public setting is the one-way analysis of variance (ANOVA), which tests the equality of means across many groups. Private ANOVA has been studied previously first by Campbell et al. [5] and then by Swanberg et al. [31], who improved the power by an order of magnitude. The standard nonparametric test in the public setting is the Kruskal-Wallis test, which was used by the pschosis research group to determine that subjects in the schizophrenia, bipolar, and control groups had different methylation levels at a particular gene site[6]. As is standard for nonparametric statistics in the public setting, it sacrifices some power compared to ANOVA but no longer assumes normally distributed data. [12]

In this section we present two tests. The first is a straightforward privatization of the standard Kruskal-Wallis test statistic. The second modifies the statistic, essentially by linearizing the implied distance metric. We find first that our modified statistic has much higher power. We then further show that our modified statistic has much higher power than the ANOVA test of Swanberg et al. even when the data is normally distributed.

The Kruskal-Wallis test

The Kruskal-Wallis test, proposed by William Kruskal and W. Allen Wallis in 1952 [20], is used to determine if several groups share the same distribution in a continuous variable. The only assumptions are that the data are drawn randomly and independently from a distribution with at least an ordinal scale.

Take a database x with $g$ groups555Throughout the paper we assume $g$ is public and independent of the data, so we don’t list it as a separate input. Because $g$ is the number of valid groups, one or more of the $g$ groups might not contain any observations. Allowing many valid groups that have no actual observations artificially increases the critical value, so it can reduce the power of our tests but does not affect the validity or privacy of the output. and $n$ rows. Let $n_{i}$ be the size of each group and $r_{ij}$ be the rank of the $j^{\text{th}}$ element of group $i$. (If values are equal for several elements, all are given a rank equal to the average rank for that set.) We define $\bar{r}_{i}$ to be $\frac{1}{n_{i}}\sum_{j=1}^{n_{i}}r_{ij}$, the mean rank of group $i$, and $\bar{r}$ to be $\frac{n+1}{2}$, the average of all the ranks. Then, the Kruskal-Wallis $h$-statistic is defined to be

[TABLE]

If there are no ties in the database, the denominator is constant and the formula can be simplified to

[TABLE]

For clarity and consistency with later sections, we present this calculation as an algorithm. In general we use a subscript “stat” to label the algorithm computing a test statistic and a subscript “p” to denote the fully hypothesis test that outputs a p-value. We use tildes to indicate private algorithms.

Privatized Kruskal-Wallis

In this section, we bound the sensitivity of $\sf{KW_{\textup{stat}}}$, allowing us to create a private version. We then present a complete algorithm for calculating a p-value and prove that it too is differentially private. We begin with the following sensitivity claim, proved in Appendix A.1.

Theorem 3.1**.**

The sensitivity of $\sf{KW_{\textup{stat}}}$ is bounded by 87.

We are using the simplified formula that assumes there are no ties in the data, so our algorithm begins by adding a small amount of random noise to each data point to randomly order any ties. We may then compute the $h$-statistic as in the public setting and add noise proportional to the sensitivity.

Theorem 3.2**.**

Algorithm $\sf{\widetilde{KW}_{\textup{stat}}}$ is $\epsilon$-differentially private.

See Appendix A.2 for the proof.

Algorithm $\sf{\widetilde{KW}_{\textup{p}}}$ is our complete algorithm to find a p-value given a database x, privacy parameter $\epsilon$. First a private test statistic $\widetilde{h}$ is computed. Then the reference distribution is approximated by simulating $z$ databases under $H_{0}$ and computing the test statistic for each.666When we use the traditional Kruskal-Wallis test, the distribution of $h$-statistics asymptotically converges to the $\chi^{2}$ distribution. Thus, for efficiency purposes, we sample $h_{k}$ from ${\sf\chi^{2}}(g-1)+\textsf{Lap}(\Delta h/\epsilon)$ (The distribution of the test statistic is independent of the distribution of data between groups and the distribution of the i.i.d. data points, so our choice of equal-sized groups and uniform data from $[0,1]$ is arbitrary.) The p-value is the percent of $h_{k}$ more extreme than $\widetilde{h}$.

Theorem 3.3**.**

Algorithm $\sf{\widetilde{KW}_{\textup{p}}}$ is $\epsilon$-differentially private.

Proof.

By Theorem 3.2, the computation of $\widetilde{h}$ is $\epsilon$-differentially private. All of the following steps (generating the reference distribution and calculating $p$-value) do not need to access to the database x, and therefore by Theorem 2.4 (post processing), Algorithm $\sf{\widetilde{KW}_{\textup{p}}}$ is $\epsilon$-differentially private. ∎

A New Test: Absolute Value Kruskal-Wallis

We now introduce our own new test, specifically designed for the private setting. Inspired by Swanberg et al. [31], we alter the Kruskal-Wallis statistic, measuring distance with the absolute value instead of the square of the differences. This statistic is now

[TABLE]

As before, when there are no ties in the data, the statistic can be simplified. (See Appendix A.3 for the calculation.) In this case, the form depends on the parity of $n$.

[TABLE]

We call the algorithm to calculate the $h_{abs}$ test statistic $\sf{KWabs_{\textup{stat}}}$. This statistic is preferable for two reasons. First, it has lower sensitivity. The following theorem is prooved in Appendix A.4.

Theorem 3.4**.**

The sensitivity of $\sf{KWabs_{\textup{stat}}}$ is bounded by 8.

Second, the actual values for $h_{abs}$ are significantly higher than they are for $h$, so any given amount of noise is less likely to overwhelm the value.

Because of space constraints, we don’t give pseudocode for this hypothesis test, but it follows exactly that of the previous test. The privatized statistic is computed by $\sf{\widetilde{KWabs}_{\textup{stat}}}$, which adds Laplace noise in the same way as for $\sf{\widetilde{KW}_{\textup{stat}}}$, but scaled to the lower sensitivity. The full hypothesis test, $\sf{\widetilde{KWabs}_{\textup{p}}}$, computes the $p$ value in the same way as was done for $\sf{\widetilde{KW}_{\textup{p}}}$.777Unlike before, a ${\sf\chi^{2}}$ approximation cannot be used.

Theorem 3.5**.**

Algorithm $\sf{\widetilde{KWabs}_{\textup{stat}}}$ and $\sf{\widetilde{KWabs}_{\textup{p}}}$ are $\epsilon$-differentially private.

Proof.

The proof is identical to the proofs for Algorithms $\sf{\widetilde{KW}_{\textup{stat}}}$ and $\sf{\widetilde{KW}_{\textup{p}}}$ (Theorems 3.2 and 3.3). ∎

Unequal group sizes

The traditional $h$ statistic (and therefore the noisy private analogue) has a reference distribution that is independent of the allocation of observations between groups. This is unfortunately not true for our new $h_{abs}$ statistic. Fortunately, it seems that the worst-case distribution (i.e., the one resulting in the highest critical value) occurs when all groups are of equal size. (We present both theoretical and experimental evidence for this in Appendix A.5.) As a result, it is safe to always equal-sized groups when simulating a reference distribution, though for very unequal group sizes, there will be a significant loss in power compared to a hypothetical where group sizes were known. (If approximate group sizes are known publicly or released through other queries, those could be used instead when simulating the reference distribution.)

Experimental Results

Power analysis

We now assess the power of our $\sf{\widetilde{KWabs}_{\textup{p}}}$ test on synthetic data (See Appendix B.1 for an application to real-world data.) We generate many databases of data distributed with specified parameters and then run $\sf{\widetilde{KWabs}_{\textup{p}}}$ on each. The power of the test for a given set of parameters is the proportion of times $\sf{\widetilde{KWabs}_{\textup{p}}}$ returns a significant result (i.e. a p-value less than the significance level $\alpha$, generally set at 0.05). We use three groups of normally-distributed data, separated by steps of one standard deviation (so the highest and lowest groups differ by two standard deviations). In our captions we denote the mean of group $i$ with $\mu_{i}$.

As shown in Figure 1, our private absolute value test variant requires significantly less data points than the original private test to reach the same power. Thus, from here on, we only evaluate the power of the absolute value variant. Figure 1 also shows that, at an $\epsilon$ of 1, our private absolute value test only requires a database around a factor of 3 larger than the public test needs.

Uniformity of p-values

If a test is correctly designed, the probability of type 1 error (i.e., rejecting the null hypothesis when it is correct) should be less than or equal to $\alpha$ for any chosen value of $\alpha$. Comparing the fit of a large number of simulated p-values generated from null distributions to the uniform distribution on the unit interval allows one to evaluate the uniformity of p-values for a given hypothesis test. A common tool to carry this procedure out, the quantile-quantile (or Q-Q) plot, plots the quantiles of the uniform, theoretical distribution against the quantiles of the p-values. The theoretical and emperical quantiles will be nearly equal at all quantiles when the p-values follow the theoretical distribution, resulting in a linear trend on the Q-Q plot. A convex Q-Q plot indicates an increase in the type II error rate (i.e. the test not rejecting the null hypothesis when it is indeed not true, causing a decrease in power) which is acceptable but undesirable, while a concave Q-Q plot indicates an exceedingly high type I error rate (i.e. the test rejecting the null hypothesis when it is true, causing undue increases in power) which is not acceptable. Figure 2 demonstrates the p-value uniformity of $\sf{\widetilde{KWabs}_{\textup{p}}}$. See Appendix B.1 for a discussion of uniformity of p-values with unequal group sizes.

Comparison to previous work

The only prior work on hypothesis testing for independence of two variables, one continuous and one categorical, is that on ANOVA. The best private ANOVA analogue is that of Swanberg et al. [31]. In Figure 3 we compare $\sf{\widetilde{KWabs}_{\textup{p}}}$ to their test and we find its power to be much greater. To get 80% power with this effect size, our test requires only 23% as much data as the private ANOVA test. (The effect size used is the same as in [31].) We stress that this means our test is significantly higher-power, in addition to being usable for non-normal data. The test of Swanberg et al. also requires that the analyst issuing the query can accurately bound the range of the data—a bound that is too tight or too loose will reduce the power of the test. Our test works for data with unknown range.

Robustness of results

Though it is unusual, it is possible that the relative power of different hypothesis tests could change when different effect sizes are considered. Therefore we repeat the analysis shown in Figure 1 with a variety of different effect sizes, group sizes, and number of groups. We also vary the frequency of tied values in the data, since the random ordering of tied values adds additional noise for our statistic. Finally, we run the comparison on real data comparing income and age. The results of these experiments are shown in Appendix B.1. We find that the results discussed above are consistent across these variations.

Two Groups

We now consider the case of data with only two groups (e.g., restricting our comparison to the methylation levels of the bipolar subjects versus the healthy controls.) In the public nonparametric setting, one could simply use Kruskal-Wallis with $g=2$, but one can also use the Mann-Whitney $U$-test (also called the Wilcoxon rank-sum test), proposed in 1945 by Frank Wilcoxon [39] and formalized in 1947 by Henry Mann and Donald Whitney [21]. In this section we construct a private version of the Mann-Whitney test and compare it to simply using $\sf{\widetilde{KWabs}_{\textup{p}}}$ with $g=2$.

The standard parametric test in the public setting is the two-sample t-test. We know of three prior works that can, in some sense, be seen as providing an analogue of the two-sample t-test for the standard private setting. The only one for which this is an explicit goal is that of D’Orazio et al. [9]. This test releases private estimates of the difference in means and of the within-group variance and produces a confidence interval rather than a p-value. (The difference in means is done with simple Laplace noise, while the variance estimate uses a subsample-and-aggregate algorithm.) Most importantly, they assume that the size of the two groups is public knowledge, where we treat the categorical value of a data point (ex., schizophrenic or not) to be private data.

There are two other works we know of that provide a private analogue of the two-sample t-test as a result of a slightly different goal. The first is Ding et al. [8], who give a test under the more restrictive local differential privacy definition. This test is of course also private under the standard differential privacy definition. The other work is that of Swanberg et al. [31], who give a private analogue of the ANOVA test, as discussed previously. In the public setting, ANOVA with $g=2$ is equivalent to the two-sample t-test. Based on (somewhat incomparable) experiments in their respective papers, it appears that the Swanberg et al. test is much higher power, which is unsurprising given that it was developed for the centralized database model of privacy. We therefore compare our work to this.

To our knowledge, there is no prior work specifically on a private version of the Mann-Whitney test. As before, we find that our rank-based nonparametric tests outperform the private parametric equivalent even when the data is normally distributed. We also find that, unlike in the public setting, the more generic Kruskal-Wallace analogue (used with $g=2$) outperforms the more purpose-built test.

The Mann-Whitney test

The function used to calculate the Mann-Whitney $U$ statistic is formalized in Algorithm $\sf{MW_{\textup{stat}}}$. As before, x is a database of size $n$, with $r_{ij}$ being the rank of the $j^{\text{th}}$ data point in group $i$. A statistic is first calculated for each group by summing the rankings in that group and subtracting a term depending on the group size. We then take the minimum of the two statistics to get $U$. Compared to the other statistics we are considering, the directionality of $U$ is reversed — low values are inconsistent with the null hypothesis and cause rejection, rather than high values.

A Differentially Private Algorithm

The global sensitivity of $\sf{MW_{\textup{stat}}}$ is $n$, but the local sensitivity is lower. We prove the following in Appendix A.6:

Theorem 4.1** (Sensitivity of $\sf{MW_{\textup{stat}}}$).**

The local sensitivity is given by $LS_{\sf{MW_{\textup{stat}}}}(\textbf{x})=\max\{n_{1},n_{2}\}$, where $n_{1}$ and $n_{2}$ are the sizes of the two groups in x.

We can now define our private test statistic, $\sf{\widetilde{MW}}_{\textup{stat}}$. This algorithm first uses a portion of its privacy budget ($\epsilon_{m}$) to estimate the size of the smallest group. This value is then reduced to $m^{*}$, such that with probability $1-\delta$ we have $n-m^{*}>LS_{\sf{MW_{\textup{stat}}}}(\textbf{x})$. This means that we can then release $U$ using noise proportional to $n-m^{*}$ (using the remaining privacy budget, $\epsilon_{U}$. See Appendix A.7 for proof that $\sf{\widetilde{MW}}_{\textup{stat}}$ is $(\epsilon_{m}+\epsilon_{U},\delta)$-differentially private.

As before, $\sf{\widetilde{MW}}_{\textup{stat}}$ is not meaningful on its own; we want an applicable reference distribution with which to calculate a corresponding $p$-value. This is shown below in algorithm $\sf{\widetilde{MW}_{\textup{p}}}$. It works similarly to the analogous algorithms $\sf{\widetilde{KW}_{\textup{p}}}$ and $\sf{\widetilde{KWabs}_{\textup{p}}}$. The key difference is that the reference distribution now depends on the group size estimate $\widetilde{m}$.888The algorithm given simulates full databases to compute the reference distribution. This is not particularly slow, but in Appendix A.8 we show that one can also sample from a normal distribution with certain parameters to get an acceptable reference distribution more quickly.

A note on design

In the case of $\sf{\widetilde{KWabs}_{\textup{p}}}$ we found that the highest possible critical value came from a reference distribution with equal-size groups. For this test that is not the case, so we cannot use equal-size groups when generating the reference distribution without unacceptable type 1 error. As a result, we need an estimate of group size. If we didn’t need this estimate for the reference distribution, it is possible that $\sf{\widetilde{MW}}_{\textup{stat}}$ would be more accurate by simply using the global sensitivity bound on $\sf{MW_{\textup{stat}}}$. (It would be a slightly higher sensitivity, but no privacy budget would need to be expended on estimating $m$.) This is a good example of a point made in Section 1: simply acheiving an accurate of estimate of a test statistic is not enough. The ultimate goal of a hypothesis test is a p-value, which also requires an accurate reference distribution and high power in order to minimize decision error.

Type 1 error

The reference distribution in the $\sf{\widetilde{MW}_{\textup{p}}}$ algorithm depends on $m$, which is only estimated by $\widetilde{m}$, so we need to experimentally verify that type 1 error never exceeds $\alpha$. See Appendix B.2 for evidence that our estimate appears to be sufficiently accurate and for additional discussion.

Theorem 4.2**.**

Algorithm $\sf{\widetilde{MW}_{\textup{p}}}$ is $(\epsilon_{m}+\epsilon_{U},\delta)$-differentially private.

Proof.

By Theorem A.6, the computation of $(\widetilde{m},\ \widetilde{U})$ is $(\epsilon_{m}+\epsilon_{U},\delta)$-differentially private. All of the steps following this computation do not require access to the database and are, thus, post processing. By Theorem 2.4, it follows that the complete algorithm is also $(\epsilon_{m}+\epsilon_{U},\delta)$-differentially private. ∎

Experimental Results

Power analysis

We first assessed the power of our test on synthetic data.999For application of our test to real-world data, see Appendix B.2 We run $\sf{\widetilde{MW}_{\textup{p}}}$ on many simulated databases and report the percentage of the time that a significant result was obtained. For our first effect size, we have the two groups consist of normally distributed data with means one standard deviation apart. In all experiments we set $\delta=10^{-6}$.

Our first step was to determine the optimal proportion of the total privacy budget, $\epsilon_{tot}$, to allot to estimating $m$ and the test statistic $\sf{\widetilde{MW}}_{\textup{stat}}$. We found that the optimal proportion of $\epsilon$ to allot to estimating $m$ is roughly $.65$, experimentally confirmed at several choices of $\epsilon_{tot}$, effect size, total sample size $n$, group size ratios $n_{1}/n$, and underlying distribution. (See Appendix B.2 for more details.) We then fix the proportion of $\epsilon_{tot}$ allotted to $\epsilon_{m}$ as $.65$ and vary $\epsilon_{tot}$ and total sample size $n$ to measure the power of our test.

As shown in Figure 4, the power loss due to privacy is not unreasonably large. At an $\epsilon_{tot}$ of $1$, the test only requires a database that is approximately a factor of $3$ larger than that needed for the public test to reach a power of $1$. As one might expect, the database size needed to detect a given effect has a roughly inverse relationship with $\epsilon_{tot}$. In the appendix we perform a similar power analysis, varying effect size rather than sample size.

Uniformity of p-values

Algorithm $\sf{\widetilde{MW}_{\textup{p}}}$ uses the privatized group sample sizes $m^{*},(n-m^{*})$ in place of the true group sizes $n_{1}$, $n_{2}$ in order to simulate the reference distribution. Naturally, then, one may wonder how conservative our critical values are as a result of ensuring that the type 1 error rate does not exceed $\alpha$. As shown in Figure 5, the type I error rate of our test does not exceed $\alpha$ when group sample sizes are equal. As total sample size $n$ increases, the p-value quantiles asymptotically approach that of the theoretical distribution. In Appendix B.2, we also examine uniformity of p-values of $\sf{\widetilde{MW}_{\textup{p}}}$ with unequal group sizes and a variation of $\sf{\widetilde{MW}_{\textup{p}}}$ that assumes equal group sizes.

Comparison to previous work

The best existing test applicable in the same use case is that of Swanberg et al. [31]. Their differentially private ANOVA test can be used in the 2-group case to compare to our Mann-Whitney test. The results of this comparison, using the same paramater settings chosen for optimal power in their test, can be seen in Figure 6, where our test offers a substantial power increase.

Comparing $\sf{\widetilde{MW}_{\textup{p}}}$ and $\sf{\widetilde{KWabs}_{\textup{p}}}$

Both the Mann-Whitney and the Kruskal-Wallis can be used to compare the distributions of samples from two groups. As shown in Figure 7, we find that in the private setting, $\sf{\widetilde{KWabs}_{\textup{p}}}$ is more statistically powerful than $\sf{\widetilde{MW}_{\textup{p}}}$. This is perhaps surprising, since one might expect the test developed specifically for the two-group case to perform better. But this is an example of how some tests privatize more easily than others. $\sf{\widetilde{MW}_{\textup{p}}}$ requires knowledge of the group sizes, using up a fraction of the privacy budget, while the $\sf{\widetilde{KWabs}_{\textup{p}}}$ statistic is not dependent on group size.

We did find one exception to this finding. If the analyst knows a priori that the two groups are of equal size (e.g., the data collection method guaranteed an equal number in each group) then $\sf{\widetilde{MW}_{\textup{p}}}$ can be run using an exact value of $n/2$ for the local sensitivity without the need to dedicate any privacy budget to estimating $m$. This increases the accuracy of $\sf{\widetilde{MW}}_{\textup{stat}}$ both by reducing the sensitivity used to add noise and by increasing the privacy budget allocated to $U$. We find that in this case $\sf{\widetilde{MW}_{\textup{p}}}$ is superior to $\sf{\widetilde{KWabs}_{\textup{p}}}$. See Appendix B.2 for more details.

Paired Data

We now consider a third situation, where there are two groups and the observations in those groups are paired. While this scenario did not exist in the original psychotic disease study, one can imagine recording the methylation levels of one of the groups before and after administering medication. Each subject then contributes a pair of data $(u_{i},v_{i})$ that are highly correlated with one another. One can assess the impact of the medication by considering whether the set of $n$ differences, $\{v_{i}-u_{i}\}_{i}$, is plausibly centered at zero. The standard nonparametric hypothesis test for this situation is the Wilcoxon signed-rank test, proposed in 1945 by Frank Wilcoxon [40]. The parametric alternative is a simple one-sample t-test run on the set of differences.

This is the one setting where we are aware of prior work on a nonparametric test. Task and Clifton [33] gave the first private analogue of the Wilcoxon signed-rank test, referred to from here on as the TC test, in 2016. Our test makes two key improvements to theirs and exhibits higher power. We also correct some errors in their work. We discuss the differences in more detail in Section 5.3.

Despite its status as one of the most commonly used hypothesis tests, to our knowledge there is no practical, implementable private version of a one-sample t-test in the literature. In Section 5.4 we discuss some work that comes close, and then we give our own first attempt at a private t-test. We again find that our nonparametric test has significantly higher power than this parametric alternative.

The Wilcoxon signed-rank test

The function calculating the Wilcoxon test statistic is formalized in Algorithm $\mathcal{W}$. Given a database x containing sets of pairs $(u_{i},v_{i})$, the test computes the difference $d_{i}$ of each pair, drops any with $d_{i}=0$, and then ranks them by magnitude. (If magnitudes are equal for several differences, all are given a rank equal to the average rank for that set.)

Under the null hypothesis that $u_{i}$ and $v_{i}$ are drawn from the same distribution, the distribution of the test statistic $W$ can be calculated exactly using combinatorial techniques. This becomes computationally infeasible for large databases, but an approximation exists in the form of the normal distribution with mean 0 and variance $\frac{n_{r}(n_{r}+1)(2n_{r}+1)}{6}$, where $n_{r}$ is the number of rows that were not dropped. Knowing this, one can calculate the p-value for any particular value of $w$.

Our Differentially Private Algorithm

At a high level, our algorithm is quite straightforward and similar to prior work: we compute a test statistic as one might in the public case and add Laplacian noise to make it private. However, there are several important innovations relative to Task and Clifton that greatly increase the power of our test.

Our first innovation is to use a different variant of the Wilcoxon test statistic. While the version introduced in Section 5.1 is the one most commonly used, other versions have long existed in the statistics literature. In particular, we look at a variant introduced by Pratt in 1959 [25]. In this variant, rather than dropping rows with $d_{i}=0$, those rows are included. When $d_{i}=0$ we set $s_{i}=\textup{Sign}(d_{i})=0$, so those rows contribute nothing to the resultant statistic, but they do push up the rank of other rows.

In the public setting, the Pratt variant is not very different from the standard Wilcoxon, being slightly more or less powerful depending on the exact effect one is trying to detect [7]. In the private setting, however, the difference is substantial.

The benefit to the Pratt variant comes from how the test statistics are interpreted. In the standard Wilcoxon, it is known that the test statistic follows an approximately normal distribution, but the variance of that distribution is a function of $n_{r}$, the number of non-zero $d_{i}$ values. In the private setting, this number is not known, and this has caused substantial difficulty in prior work. (See Section 5.3 for more discussion.) On the other hand, the Pratt variant produces a test statistic that is always compared to the same normal distribution, which depends only on $n$. The algorithm $\sf{\widetilde{WP}_{\textup{stat}}}$ that outputs a differentially private analogue is shown below.

Theorem 5.1**.**

Algorithm $\sf{\widetilde{WP}_{\textup{stat}}}$ is $\epsilon$-differentially private.

See Appendix A.9 for proof of Theorem 5.1.

To complete the design of our test, we compute a reference distribution through simulation as was done in $\sf{\widetilde{KWabs}_{\textup{p}}}$ and $\sf{\widetilde{MW}_{\textup{p}}}$. Here we use the standard normal approximation for the distribution of the $w$ test statistic, though one could simulate full databases as well. We call this algorithm $\sf{\widetilde{WP}_{\textup{p}}}$.

Theorem 5.2**.**

Algorithm $\sf{\widetilde{WP}_{\textup{p}}}$ is $\epsilon$-differentially private.

Proof.

The computation of $\widetilde{w}$ was already shown to be private. The remaining computation needed to find the p-value does not need access to the database—it is simply post-processing. By Theorem 2.4, it follows that the $\sf{\widetilde{WP}_{\textup{p}}}$ algorithm is also private. ∎

Experimental Results

Power analysis

We assess the power of our differentially-private Wilcoxon signed-rank test first on synthetic data. (For tests with real data, see Appendix B.3.) In order to measure power, we must first fix an effect size. We chose to have the $u_{i}$ and $v_{i}$ values both generated according to normal distributions with means one standard deviation apart. We then measure the statistical power of Algorithm $\sf{\widetilde{WP}_{\textup{p}}}$ (for a given choice of $n$ and $\epsilon$) by repeatedly randomly sampling a database x from that distribution and then running $\sf{\widetilde{WP}_{\textup{p}}}$ on that database.101010Our actual implementation differs slightly from this. To save time when running a huge number of tests with identical $n$ and $\epsilon$, we first generate the reference distribution $W_{k}$ values, which can be reused across runs. The power is the percentage of the time $\sf{\widetilde{WP}_{\textup{p}}}$ returns a p-value less than $\alpha$. See Appendix B.3 for a similar analysis of power, varying effect size rather than sample size.

Uniformity of p-values

In algorithm $\sf{\widetilde{WP}_{\textup{p}}}$ we draw our reference distribution samples (the $w_{k}$ values) assuming there are no $d_{i}=0$ rows. The distribution will technically differ slightly when there are many rows with $d_{i}=0$, so we need to confirm experimentally that the difference is inconsequential or otherwise acceptable.

Figure 9 shows a Q-Q plot of $\sf{\widetilde{WP}_{\textup{p}}}$ on three sets of p-values, all generated under $H_{0}$, with $\epsilon=1$, $n=500$. When there are no ties in the original data (0% of $d_{i}=0$), the Q-Q plot line is indistinguishable from the identity line, indicating that the test is properly calibrated. Encouragingly, introducing a substantial number of ties into the data (30% of $d_{i}=0$) has little noticeable effect. In order to induce non-uniformity in the p-values, one needs an extremely high proportion of rows with $d_{i}=0$. The curve with 90% zero values is shown as an illustration. When the proportion of zeros is very high, the variance of the p-values will be narrower than the reference distribution, resulting in a lower critical value. Since the value we are using is higher, our test is overly conservative,111111One could try to estimate the number of zeros to be less conservative, but that would require allocating some of the privacy budget towards that estimate, which is not worth it in most circumstances. but this is acceptable as type I error is still below $\alpha$.

Comparison to previous work

In 2016, Task and Clifton [33] introduced the first differentially private version of the Wilcoxon signed-rank test, from here on referred to as the TC test. Our work improves upon their test in two ways. We describe the two key differences below, and then compare the power of our test to theirs. We also found a significant error in their work.121212This error has been confirmed by Task and Clifton in personal correspondence. All comparisons are made to our implementation of the TC test with the relevant error corrected.

Task and Clifton compute an analytic upper bound on the critical value $t^{*}$. For a given $n$ and $\epsilon$, the private test statistic $\widetilde{w}$ under $H_{0}$ is sampled according to a sum $W+\Lambda$, where $W$ is a random draw from a normal distribution (scaled according to $n$) and $\Lambda$ is a Laplace random variable (scaled according to $n$ and $\epsilon$). In particular, say that $b$ is a value such that $\Pr[W>b]<\beta$ and $g$ is a value such that $\Pr[\Lambda>g]<\gamma$. Then we can compute the following bound. (The last line follows from the fact that the two events are independent.)

[TABLE]

Task and Clifton always set $\gamma=.01$ and then vary the choice of $\beta$ such that they have $\alpha=\beta+\gamma-\beta\gamma$ for whatever $\alpha$ is intended as the significance threshold.131313This is where Task and Clifton make an error. This formula is correct, but they used an incorrect density function for the Laplace distribution and as a result calculated incorrect values of $g$.

The bound described above is correct but very loose, and our simulation method gives drastically lower critical values. Table 1 contains examples of the critical values achieved by each method for several parameter choices. More values can be found in Appendix B.3, where we also experimentally confirm that these values result in acceptable type 1 error.

Our second key change from the TC test, mentioned earlier, is that we handle rows with $d_{i}=0$ according to the Pratt variant of the Wilcoxon, rather than dropping them completely as is more traditional. The reason the traditional method is so difficult in the private setting is that the reference distribution one must compare to depends on the number of rows that were dropped. If $n_{r}$ is the number of non-zero rows (i.e., rows that weren’t dropped), one is supposed to look up the critical value associated with $n_{r}$, rather than the original size $n$ of the database.

Unfortunately, $n_{r}$ is a sensitive value and cannot be released privately.141414A private estimate could be released, but one would have to devote a significant portion of the privacy budget for the hypothesis test to this estimate, greatly decreasing the accuracy/power of $\sf{\widetilde{W}_{\textup{stat}}}$. Task and Clifton show that it is acceptable (in that it does not result in type I error greater than $\alpha$) to compare to a critical value for a value of $n_{r}$ that is lower than the actual value. This allows them to give two options for how one might deal with the lack of knowledge about $n_{r}$.

High Utility

This version of the TC test simply assumes $n_{r}\geq.3n$ and uses the critical value that would be correct for $n_{r}=.3n$. We stress that this algorithm is not actually differentially private, though it could easily be captured by a sufficiently weakened definition that limited the universe of allowable databases. Another problem is that for most realistic data, $n_{r}$ is much greater than $.3n$ and using this loose lower bound still results in a large loss of power.

High Privacy

This version adds $k$ dummy values to the database with $d_{i}=\infty$ and $k$ with $d_{i}=-\infty$.151515Task and Clifton do not discuss how to choose $k$, and in our experimental comparisons we set $k=15$, the same value they use. Then one can be certain of the bound $n_{r}\geq 2k$. This is a guaranteed bound so this variant truly satisfies differential privacy. On the other hand it is a very loose lower bound in most cases, leading to a large loss of power.

Experimental comparison

We compare the statistical power of our test to that of the TC test. We begin by again measuring the power when detecting the difference between two normal distributions with means one standard deviation apart. The results can be seen in Figure 10. If we look at the database size needed to achieve 80% power, we find that the 32 data points we need, while more than the public test (14), are many fewer than the TC High Utility variant (80) or the TC High Privacy variant (122). Appendix B.3 includes a figure for $\epsilon=.1$ as well. What we see is that, while all private tests require more data, our test (requiring $n\approx 236$) still requires about 40% as much data as the TC High Utility variant (588). The TC High Privacy variant, however, scales much less well to low $\epsilon$ and requires roughly 2974 data points.

The results in Figure 10 use a continuous distribution for the real data, so there are no data points with $d_{i}=0$. Because one of the crucial differences between our algorithms is the method for handling these zero values, we also consider the effect when there are a large number of zeros in Appendix B.3. Overall, we see that both in situations with no zero values and situations with many, our test achieves the rigorous privacy guarantees of the TC High Privacy test while achieving greater utility than the TC High Utility test.

Relative contribution of improvements

Given that we make two meaningful changes to the TC test, one might naturally wonder whether both are truly useful or whether the vast majority of the improvement comes from one of the two changes. To test this, we compare to an updated variant of the TC test where we calculate critical values exactly through simulation, as we do in our algorithm, but otherwise run the TC test unchanged (referred to as ”High Privacy +” and ”High Utility +”). The result is presented in Figure 11, where we find the resulting algorithm to rest comfortably between the original TC test and our proposed test. This means that both the change to the critical value calculation and the switch to the Pratt method of handling $d_{i}=0$ rows are important contributions to achieving the power of our test.

Parametric Alternative: A New T-test

The parametric analog to the Wilcoxon test is to run a one sample t-test on the set of differences $\{v_{i}-u_{i}\}_{i}$ to see if their mean is significantly different from zero (also called a paired t-test). There has been surprisingly little work on the creation of a private version of a one sample t-test. Karwa and Vadhan [19] study private confidence intervals, which are in a sense equivalent to a t-test. However, their analysis is asymptotic and they say that the algorithm does not give practical results with database size in the thousands. Sheffet [27] provides a method for calculating private coefficient estimates for linear regression and also transforms the t-distribution to provide an appropriate reference distribution for inference. In the public setting, one can convert a test on regression coefficients to a one sample t-test but choosing a constant independent variable and making the sample data the dependent variable. However, Sheffet’s method only works when all variables are significantly spread out, so this method fails.

Here we propose what we believe is the first private version of a one sample t-test, with two arguable exceptions. The first is simultaneous work by Gaboardi et al. [15] in the local privacy model. We compare our results to theirs in more detail in Section 5.5. The other work is that of Solea [30], but according to Solea’s own experiments that test often gives type 1 error rates well above the chosen $\alpha$ for many parameter choices, so we don’t consider it a usable test.

The database for a one sample t-test has observations $x_{1},\ldots,x_{n}$ assumed to come from a normal distribution with mean $\mu$ and standard deviation $\sigma$. (For paired data, each observation is the differences between the observation in the two groups). The test statistic is given by $\sf{T_{\textup{stat}}}(\textbf{x})=\frac{\bar{x}}{s/\sqrt{n}}$, where $\bar{x}$ is the mean of the data and $s$ is the standard deviation of the data.

A private t-test

As before, we achieve privacy through the addition of Laplacian noise, but the sensitivity of $\sf{T_{\textup{stat}}}$ is unbounded, so we instead release separate private estimates of the numerator and denominator. For this analysis, similar to the private ANOVA tests [31], we assume that the data is scaled such that all observations are on the interval $[-1,1]$. We first find the sensitivities of $\bar{x}$ and $s^{2}$ and then use post-processing, composition, and the Laplace Mechanism to combine these to obtain the private t-statistic. In the case where $s^{2}$ is estimated to be negative, the test statistic cannot be computed as normal, and we return 0, indicating an unwillingness to reject the null hypothesis.

Theorem 5.3**.**

The sensitivity of $\bar{x}$ is $\frac{2}{n}$ .

Theorem 5.4**.**

The sensitivity of $s^{2}$ is $\frac{5}{n-1}$.

See Appendix A.10 for proof of Theorem 5.3 and 5.4.

Theorem 5.5**.**

Algorithm $\sf{\widetilde{T}_{\textup{stat}}}$ is $(\epsilon_{\bar{x}}+\epsilon_{s^{2}})$-differentially private.

Proof.

By the Laplace mechanism, the computation of $\widetilde{\bar{x}}$ is $\epsilon_{\bar{x}}$-differentially private and the computation of $\widetilde{s^{2}}$ is $\epsilon_{s^{2}}$-differentially private. Since the computation of $\widetilde{T}$ does not require access to the database, it is only post-processing and its release is $(\epsilon_{\bar{x}}+\epsilon_{s^{2}})$-differentially private. ∎

To carry out the full paired t-test, we estimate the reference distribution through simulation and release a private p-value.

Theorem 5.6**.**

Algorithm $\sf{\widetilde{T}_{\textup{p}}}$ is $\epsilon_{\bar{x}}+\epsilon_{s^{2}}$-differentially private.

Proof.

The computation of $\widetilde{t}$ was already shown to be private. The remaining computation needed to find the p-value does not need access to the database—it is simply post-processing. By Theorem 2.4, it follows that the $\sf{\widetilde{T}_{\textup{p}}}$ algorithm is also private. ∎

Experimental t-Test evaluation

We first must set a parameter in our $\sf{\widetilde{T}_{\textup{p}}}$ algorithm. In particular, for a given total $\epsilon$, we must decide how to allocate the budget between $\epsilon_{\bar{x}}$ and $\epsilon_{s^{2}}$. We choose this allocation experimentally, deciding to allocate 50% of the budget towards each value. This is nontrivial, and Appendix B.4 contains experimental results and further discussion. Luckily, the exact choice of this allocation does not seem to have a large effect on the power of the test.

We then evaluate the power and validity of the final $\sf{\widetilde{T}_{\textup{p}}}$ test.

Comparison to other work

Simultaneous to our work, Gaboardi et al. [15] developed a private one sample t-test under the more restrictive local differential privacy model. As one might expect, our test in the more standard setting is much higher power. They develop both a t-test and a z-test, which is equivalent to the t-test except that the variance of the data is assumed to be already known. Only the z-test is given experimental evaluation, but with an effect size three times the size we use in our experiments, their test (at $\epsilon=1$) requires roughly 4000 data points to reach 80% power, while our test requires roughly 100. Their t-test would presumably require even more data.

Comparison to nonparametric test

Since we have already developed a test for the paired-data use case, we assessed the power of $\sf{\widetilde{T}_{\textup{p}}}$ in comparison to $\sf{\widetilde{WP}_{\textup{p}}}$ by simulating synthetic data as described in Section 5.3. Just as in the many groups and two groups scenarios, the nonparametric test substantially outperforms its parametric counterpart, as shown in Figure 12. In this case, $\sf{\widetilde{WP}_{\textup{p}}}$ needs 8% of the data required by $\sf{\widetilde{T}_{\textup{p}}}$ to reach the same power.

Uniformity of p-values

As with all of our tests, we experimentally ensure that type I error rate is bounded by $\alpha$ in Figure 13. This figure confirms the fact that our type I error rate is bounded above by $\alpha$. For small sample sizes, the line on the quantile-quantile plot goes above the diagonal. This is the acceptable direction, the sign of a conservative test. In this case it occurs because some test statistics in the reference distribution are set to zero (as a result of noise added for privacy overwhelming $\widetilde{s^{2}}$). If, for example, 10% of the reference distribution samples are at zero, then p values below 10% are impossible. As shown by the $n=1000$ line, at sufficiently large sample sizes this effect essentially vanishes.

Conclusion

We have introduced several new tests, of which three ($\sf{\widetilde{KWabs}_{\textup{p}}}$, $\sf{\widetilde{MW}_{\textup{p}}}$, and $\sf{\widetilde{WP}_{\textup{p}}}$) are improvements on the state of the art. These allow researchers to address inferential questions using nonparametric methods while preserving the privacy of the data. More broadly, we found that the basic idea of using ranks in the private setting is potent. Not only do they remove the need to assume a bound on the data, they also directly increase statistical power. When working with many groups, two group, or with paired data, rank-based tests are more powerful than their parametric analogues and can be made yet more powerful through sensible adaptations. We hope others will push this technique forward — we have no reason to believe that our tests are optimal.

Acknowledgments

We would like to thank Christine Task and Chris Clifton for generous and enlightening discussions regarding their previous work. This material is based upon work supported by the National Science Foundation under Grant No. SaTC-1817245 and the Richter Funds.

Appendix A Supplementary Proofs

Proof of Theorem 3.1

We begin by introducing the following lemma and corollaries, which will be useful in the proof.

Lemma A.1**.**

For integers $a$ and $b$:

[TABLE]

Corollary A.2**.**

For group $i$ which has $n_{i}$ elements with rankings $r_{i1},\ldots,r_{in_{i}}$, we can bound the sum of the ranks by:

[TABLE]

Proof.

The maximum occurs when all the terms in the sum are as high as possible. I.e., they are $n-n_{i}+1,n-n_{i}+2,\ldots,n-1,n$. Thus,

[TABLE]

The minimum occurs when all the terms in the sum are as low as possible. I.e., they are $1,2,\ldots,n_{i}-1,n_{i}$. Thus,

[TABLE]

∎

Corollary A.3**.**

Suppose there are $g$ groups in the database, then $\forall\;1\leq i\leq g$,

[TABLE]

where $n_{i}$ is the group size.

Proof.

We define $\bar{r}_{i}$ to be $\frac{1}{n_{i}}\sum_{j=1}^{n_{i}}r_{ij}$. The proof follows directly from this definition. ∎

We now prove the main theorem:

Theorem**.**

The sensitivity of $\sf{KW_{\textup{stat}}}$ is bounded by 87.

Proof.

The proof proceeds as a series of cases. Neighboring databases $\textbf{x},\ \textbf{x}^{\prime}$ differ by a change in one row, and this difference will cause either the row’s rank, group, or both to change. The first case bounds the change in the test statistic when only the row’s rank changes. The second case is when the row’s group changes. Because our algorithm first randomly breaks any ties, we assume there are no tied values in the database when $h$ is computed.

**Case 1: **Changing row retains group

Consider two databases that differ only in that one row is “moved” from rank $p$ to rank $q$. Let us assume without loss of generality that the row is an element of group 1, so it follows that $n_{1}\geq 1$. We define the distance between the rows to be $d=|q-p|$ and $a_{i}$ to be the number of elements in group $i$ between row $p$ and row $q$ (excluding row $p$ and including row $q$).

[TABLE]

The final step follows by the triangle inequality. Let us define $z_{i}$ to be

[TABLE]

The “movement” of the row causes all of the elements of group 1 between $p$ and $q$ to change in the opposite direction of the movement (if $q>p$, then all the ranks decrease by 1, while if $q<p$, then all the ranks increase by 1). This yields a total change in the rankings of group 1 of $d-a_{1}$. Thus,

[TABLE]

Similarly,

[TABLE]

Then,

[TABLE]

Thus with only a row change, $\Delta h=48$.

**Case 1: **Changing row changes group

Consider two databases that differ only in that one row’s rank is changed from $p$ to $q$ and its group is changed from, without loss of generality, group 1 to group 2.

We define $a_{i}$ as previously. We note again that the sign of $a_{i}$ depends on the relationship between $p$ and $q$. If $q>p$, then the sign is negative; if $q<p$, then the sign is positive.

We begin by introducing a lemma we will use throughout this argument:

Lemma A.4**.**

We can bound this summation as follows:

[TABLE]

Proof.

[TABLE]

∎

When we change the group of the moving row, we must consider the sensitivity in each of three sub-cases:

Case 2a: $n_{1}>1,\ n_{2}>0$

[TABLE]

Let us define $z_{i}$ to be

[TABLE]

When the expression inside the absolute value is less than zero, $\Delta h$ can be written as

[TABLE]

Then we can bound this quantity as follows.

[TABLE]

Through differentiation of the first term with respect to $n_{1}$, we get that the maximum occurs when $n_{1}=\frac{n+2}{2}$. This gives that the first term is bounded by $\frac{9n^{2}}{4}.$ We then differentiate the second term with respect to $n_{1}$ and get that the slope is always negative, which means that the maximum of the second term maximum occurs when $n_{1}=2$. Then the second term is bounded by $2n^{2}+n-1$.

Therefore,

[TABLE]

When $i=2$:

[TABLE]

It is clear that the first term is bounded by $n^{2}$. Then for the other term, we can take the derivative with respect to $n_{1}$, which gives that the maximum occurs when $n_{2}=n+1$. But as $n_{2}\leq n$ and the graph is concave down, we can use $n_{2}=n$. Thus, that term is bounded by $(n-1)(2n-n+1)=n^{2}-1.$

Therefore,

[TABLE]

When $i>2$, we consider the sum of them:

[TABLE]

Then combining them together, it follows that

[TABLE]

When the expression inside the absolute value bars is greater than zero, we can bound this quantity as follows:

[TABLE]

Then consider $-p+a_{1}$. As it is $+a_{1}$, it matches the case $p>q$, and thus $-p+a_{1}$ is negative and we can drop it directly.

It follows that

[TABLE]

Similarly,

[TABLE]

Consider $(n+n_{2})(2n-n_{2}+1)$. The derivative with respect to $n_{2}$ is $1+n-2n_{2}$. As the graph of the equation is concave down, when the derivative is [math], it reaches its maximum. Thus the maximum occurs when $n_{2}=\frac{n+1}{2}$, which leads to the maximum $\frac{(3n+1)^{2}}{4}$.

Then it follows that,

[TABLE]

The rest of the terms, by Lemma A.4, are bounded by $n^{2}-2n$. It follows that

[TABLE]

Case 2b: $n_{1}=1,\ n_{2}>0$

[TABLE]

Now consider the term $|(n_{2}+1)\bar{r^{\prime}}_{2}^{2}-n_{2}\bar{r}_{2}^{2}|$,

[TABLE]

It follows that

[TABLE]

Case 2c: $n_{2}=0,\ n_{1}>1$

[TABLE]

Now consider the term $|(n_{1}-1)\bar{r^{\prime}}_{1}^{2}-n_{1}\bar{r}_{1}^{2}|$,

[TABLE]

note that $-p\pm a_{1}$ is always negative. If the sign is $-$, then this directly follows. If the sign is $+$, then $p>q$, which indicates $p>a_{1}$ and thus it again follows that the term is negative.

Then if the whole inside expression is negative,

[TABLE]

Then if the whole inside expression is positive,

[TABLE]

As $n\geq 2$, then $\frac{3n^{2}}{2}-3n+\frac{1}{2}\geq 0$. It follows that $\frac{9n^{2}}{2}+3n+\frac{1}{2}\geq 3n^{2}+6$, and thus

[TABLE]

Therefore,

[TABLE]

∎

Proof of Theorem 3.2

Theorem**.**

Algorithm $\sf{\widetilde{KW}_{\textup{stat}}}$ is $\epsilon$-differentially private.

Proof.

There are two sources of randomness inside Algorithm $\sf{\widetilde{KW}_{\textup{stat}}}$: the random breaking of ties and the Laplace noise. So let us write $\sf{\widetilde{KW}_{\textup{stat}}}(\textbf{x}):=\sf{\widetilde{KW}_{\textup{stat}}}(\textbf{x};r_{1},r_{2}),$ where $r_{1}$ indicates the randomness of breaking ties and $r_{2}$ indicates the randomness of the Laplace noise.

For an arbitrary choice of $r_{1}^{*}$, let $\sf{\widetilde{KW}_{\textup{stat}}}^{r_{1}^{*}}(\textbf{x};r_{2})$, be the function obtained by fixing $r_{1}=r_{1}^{*}$ in $\sf{\widetilde{KW}_{\textup{stat}}}$. $\sf{\widetilde{KW}_{\textup{stat}}}^{r_{1}^{*}}$ is simply a Laplace mechanism, so $\epsilon$-differential privacy follows by Theorems 3.1 and 2.8.

Now $\sf{\widetilde{KW}_{\textup{stat}}}$ is simply randomly outputting one of a large (but finite) set of possible $\epsilon$-differentially private outputs. Doing so is always private, so $\sf{\widetilde{KW}_{\textup{stat}}}$ is private. ∎

Simplification of $h_{abs}$

Changing the squares to absolute values in the Kruskal-Wallis yields the formula:

[TABLE]

When there are no tied rows in the data, this formula can be simplified a bit. The simplification depends on whether $n$ is even or odd.

When $n$ is even:

[TABLE]

Thus, the simplified form of $h_{abs}$ for even $n$ is:

[TABLE]

When $n$ is odd

[TABLE]

Thus, the simplified form of $h_{abs}$ for odd $n$ is:

[TABLE]

As $\frac{4}{n+1}\geq\frac{4(n-1)}{n^{2}}$, we will use $\tilde{h}=\tilde{h}_{O}=\frac{4}{n+1}\sum_{i=1}^{g}n_{i}|\bar{r}_{i}-\bar{\bar{r}}|$ for sensitivity analysis.

Proof of Theorem 3.4

Proof.

Here we bound the sensitivity of our $\sf{KWabs_{\textup{stat}}}$ test statistic.

Theorem**.**

The sensitivity of $\sf{KWabs_{\textup{stat}}}$ is bounded by 8.

As $\frac{4}{n+1}\geq\frac{4(n-1)}{n^{2}}$ for $n>0$, we will use

$h_{abs}=\frac{4}{n+1}\sum_{i=1}^{g}n_{i}|\bar{r}_{i}-\bar{\bar{r}}|$ for sensitivity analysis. As in the proof of Theorem 3.1, the proof proceeds in cases. Again, we consider neighboring databases $\textbf{x},\ \textbf{x}^{\prime}$ that differ by a change in one row, and our cases are differentiated by whether or not the group of that row changes.

**Case 1: **Changing row retains group

Consider a “movement” of a row from row $p$ to row $q$. Assume $p\leq q$ (the other case is symmetric).

[TABLE]

Then we can analyze the expression inside the outer absolute value.

When $i=1$,

[TABLE]

When $i\neq 1$,

[TABLE]

It follows that,

[TABLE]

**Case 2: **Changing row changes group

Consider a “movement” of a row from row $p$ to row $q$, where, without loss of generality, its group changes from group 1 to group 2. As before, we must consider three cases:

Case 2a: $n_{1}>1,\ n_{2}>0$

[TABLE]

For $i=1$,

[TABLE]

For $i=2$,

[TABLE]

For $i>2$,

[TABLE]

Thus, we can bound the sensitivity by:

[TABLE]

Case 2b: $n_{1}=1,\ n_{2}>0$

As before,

[TABLE]

For $i=1$,

[TABLE]

The $i\neq 1$ cases are as in case I. Thus,

[TABLE]

Case 2c: $n_{2}=0,\ n_{1}>1$

As before,

[TABLE]

For $i=2$,

[TABLE]

For $i\neq 2$, it is the same as in case II. Thus,

[TABLE]

∎

Type I Error Rate of $\sf{\widetilde{KWabs}_{\textup{stat}}}$

When we compute the reference distribution for $h_{abs}$ in the public setting, we require the size of each group to be known. In the private setting, however, the group sizes are not public and therefore cannot be used in the reference distribution. Thus, we must determine the worst case group sizes and use these to compute the reference distribution in Algorithm $\sf{\widetilde{KW}_{\textup{p}}}$.

Theorem A.5**.**

Let $H_{abs}$ be the random variable of which $h_{abs}$ is an instance. Then the expected value of $H_{abs}$ is maximized when the sizes of the groups are $n/g$.

Proof.

Let $\bar{R}_{i}$ be the random variable of which $\bar{r}_{i}$ is an instance. The distribution of $\bar{R}_{i}$ is known to be $\bar{R}_{i}\sim\mathcal{N}\left(\mu=\frac{n+1}{2},\sigma^{2}=\frac{n^{2}-1}{12n_{i}}\right)$. Let $Z_{i}$ be a standard normal random variable: $Z_{i}\sim\mathcal{N}\left(\mu=0,\sigma^{2}=1\right)$. We can relate $\bar{R}_{i}$ and $Z_{i}$ by

[TABLE]

.

Additionally, it is known that a half normal distribution from a standard normal, $|Z_{i}|$, has mean $\sqrt{\frac{2}{\pi}}$ and variance $1-\frac{2}{\pi}$.

We will use the formulation of $H_{abs}$ when $n$ is odd. The algebra is identical for an even $n$.

[TABLE]

Let us consider the expected value of $H_{abs}$:

[TABLE]

The expected value of $H_{abs}$ is maximized when $\sum_{i=1}^{g}\sqrt{n_{i}}$ is maximized. This will occur when the sizes of each of the groups are $n/g$. ∎

Proof of Theorem 4.1

Recall that

[TABLE]

where

[TABLE]

We now prove the following:

Theorem** (Sensitivity of $\sf{MW_{\textup{stat}}}$).**

The local sensitivity is given by $LS_{\sf{MW_{\textup{stat}}}}(\textbf{x})=\max\{n_{1},n_{2}\}$, where $n_{1}$ and $n_{2}$ are the sizes of the two groups in x.

Proof.

We bound the sensitivity of $U_{1}$ and $U_{2}$, and note that this bound also applies to their minimum. We consider two cases, one where the row that differs between x and $\textbf{x}^{\prime}$ retains its group and where it changes its group.

“Changing” row retains group

Consider two databases x and $\textbf{x}^{\prime}$ that differ only in row $j$ and where row $j$ is in the same group in both x and $\textbf{x}^{\prime}$. Without loss of generality, suppose $j$ is in group $1$ and that $v_{j}^{\prime}>v_{j}$ (where $v_{j}$ and $v_{j}^{\prime}$ are the values of row $j$ in x and $\textbf{x}^{\prime}$ respectively). Suppose that the row was originally (in x) part of a tie in ranks with $b$ other rows and it becomes part of a tie in ranks with $d$ other rows (in $\textbf{x}^{\prime}$). Let the number of rows between these two groups of ties be $c$ and let the number of rows before the first tie be $a$ and the number of rows after the second tie be $e$. This setup is shown in Figure 14.

The rank of $v_{j}$ in x is $\frac{a+1+a+b}{2}$ and the rank of $v_{j}^{\prime}$ in $\textbf{x}^{\prime}$ is $\frac{a+b+c+a+b+c+d}{2}$. Note that when an element is removed from a tie, the rank of every element in the tie will change by $\frac{1}{2}$.

Let $b_{1}$ and $b_{2}$ be the number of the $b$ elements (i.e., those tied with row $j$) in database x that are in groups 1 and 2, respectively. Let $c_{1}$, $c_{2}$, $d_{1}$, and $d_{2}$ be similiarly defined for the regions of $c$ and $d$ elements.

We find that the sensitivity, $LS_{U_{1}}$, is the sum of five effects. The first is the removal of the row in x, which decreases the sum by $\frac{a+1+a+b}{2}$. The second is the addition of the row in $\textbf{x}^{\prime}$, which increases the sum by $\frac{a+b+c+a+b+c+d}{2}$. The third is the effect on the rank of the $b-1$ tied elements (excluding row $j$ itself), which causes the sum to decrease by $\frac{b_{1}-1}{2}$. The fourth is the effect on the $d$ tied elements, which causes the sum to decrease by $\frac{d_{1}}{2}$. The fifth is decrease in rank for the $c$ elements that are now below instead of above row $j$, which causes the sum to decrease by $c_{1}$.

[TABLE]

Similarly, considering how ranks of group $2$ elements get changed, we get

[TABLE]

The proof is identical for the movement of a row in group 2 and yields a bound of $n_{1}$.

”Changing” row switches groups

Now let us consider x and $\textbf{x}^{\prime}$ such that one row is moved to a new group. Without loss of generality, we assume the row moves from group $1$ to group $2$. We also assume as before that $v_{j}^{\prime}>v_{j}$ and we define $a$, $b$, $c$, $d$ and $e$ as before.

For $U_{1}$, the removal of the element itself from group $1$ first causes the rank sum to decrease by $\frac{a+1+a+b}{2}$. The decrease in $n_{1}$ causes the second term of $U_{1}$ change from $\frac{n_{1}(n_{1}+1)}{2}$ to $\frac{(n_{1}-1)n_{1}}{2}$, which leads to the contribution to $LS_{U_{1}}$ to be $-\frac{(n_{1}-1)n_{1}}{2}+\frac{n_{1}(n_{1}+1)}{2}$. The rest of the effects on $U_{1}$ is the same as in the previous part. Thus,

[TABLE]

If the expression inside the absolute value is positive, it follows that

[TABLE]

If the expression inside the absolute value is negative, it follows that

[TABLE]

Then similarly for $U_{2}$,

[TABLE]

If the expression inside the absolute value is positive, it follows that

[TABLE]

If the expression inside the absolute value is positive, it follows that

[TABLE]

The proof is identical for the movement of a row in group 2 and yields a bound of the opposite row’s group size in each case. The proof is symmetric for the movement to a lower row and yields the same bound as the movement to a higher row case.

Thus, the overall sensitivity is bounded by

[TABLE]

∎

Privacy of $\sf{\widetilde{MW}}_{\textup{stat}}$

Theorem A.6**.**

$\sf{\widetilde{MW}}_{\textup{stat}}(\textbf{x})$* is $(\epsilon_{m}+\epsilon_{U}$, $\delta$)-differentially private.*

Proof.

We first divide $\sf{\widetilde{MW}}_{\textup{stat}}$ into two steps. First a function $f$ outputs $\widetilde{m}$. Then, for a given value of $\tilde{m}$, a function $g$ outputs $\widetilde{U}$. We let $g_{w}$ be the function $g$ when we have fixed $\widetilde{m}=w$. We consider an arbitrary output set $S\in\mathbb{R}^{2}$ and let $B$ be the set of outputs where $n-m^{*}$ was not a correct bound on the local sensitivity of $U$. That is, $B=\{(w,z)\mid w>m+c\}$.

First we show that output in $B$ (i.e., a bad bound for $m$) occurs only with probability $\delta$.

Lemma A.7**.**

$\Pr[\sf{\widetilde{MW}}_{\textup{stat}}(\textbf{x})\in B]\leq\delta$* .*

Proof.

First, recall that by definition $\sf{\widetilde{MW}}_{\textup{stat}}(\textbf{x})\in B$ exactly when $f(\textbf{x})>m+c$. All that remains is a simple calculation using the cumulative density function of the Laplace distribution.

Let $F(t)$ be the cumulative density function of the Laplace distribution. Then,

[TABLE]

∎

As a result of Lemma A.7, we can restrict ourselves to $S\setminus B$. Conditioned on the output being in $S\setminus B$, we know that the noise added to $U$ is sufficient to maintain privacy. Below we write $\Pr[f(\textbf{x})=w]$ to denote the probability density function of $f(\textbf{x}$) at $w$ (and analogously for $g$).

[TABLE]

Equation 4 isn’t technically an application of Theorem 2.8, but it follows from an almost identical argument. (The values of $f$ and $g_{w}$ before the addition of Laplace noise differ by a bounded amount and the noise added is scaled so that the bound on the ratio follows immediately from the formula for the Laplace distribution.)

∎

Normal approximation for $\sf{\widetilde{MW}_{\textup{p}}}$

Here we prove the acceptability of a normal approximation that increases the efficiency of computing the reference distribution for $\sf{\widetilde{MW}_{\textup{p}}}$.

Theorem A.8**.**

Using {\sf Normal}(\frac{m^{*}(n-m^{*})}{2},\frac{m^{*}(n-m^{*})(n+1)}{12})+\textsf{Lap}\Big{(}\frac{n-m^{*}}{\epsilon_{U}}\Big{)} to generate samples $U_{k}$ from the reference distribution will only make the type I error smaller compared to a full simulation.

Proof.

There are two parts that need to be proved in order to prove the theorem, and we consider them separately.

Initially, we need to show that it is acceptable for the reference distribution to be computed based on the assumption that there are no ties in the database.

When $n$ becomes very large, $U$ approximately follows a normal distribution with $\mu_{U}=\frac{n_{1}n_{2}}{2}$ and \sigma_{U}=\sqrt{\frac{n_{1}n_{2}}{12}\Big{(}(n+1)-\sum_{i=1}^{k}\frac{t_{i}^{3}-t_{i}}{n(n-1)}\Big{)}}, where $n_{1}$ and $n_{2}$ are the group sizes, $k$ is the number of distinct ranks that are tied, and $t_{i}$ is the number of rows sharing rank $i$ [21].

However, in the private setting, both $k$ and $t_{i}$ are private information and cannot be used to generate the null distribution. Since the standard deviation when there are no ties, $\sigma^{\prime}_{U}=\sqrt{\frac{n_{1}n_{2}}{12}(n+1)}$, is greater than $\sigma_{U}$,

[TABLE]

where $Z,\ Z^{\prime}$ are normal random variables when there are ties and no ties, respectively, $z^{*},\ z^{\prime*}$ are critical values for the distributions of $Z$ and $Z^{\prime}$, respectively.

Thus, according to the relation above, we can use the normal approximation as if there were no ties in the null distribution.

The second part is that, when we generate the null distribution, we need $m$ to be used for the normal distribution. However, in the differential privacy setting, $m$ indicates one of the group sizes and is private information, so it cannot be used directly to generate the private null distribution. Instead, we use $m^{*}$ to replace $m$, but we need to ensure that this substitution does not increase the type I error rate.

By previous discussion, we know that with $1-\delta$ probability, $m^{*}\leq m$. Then compare Norm$(\frac{m^{*}(n-m^{*})}{2},\frac{m^{*}(n-m^{*})(n+1)}{12})$ and

Norm$(\frac{m(n-m)}{2},\frac{m(n-m)(n+1)}{12})$. As $m^{*}(n-m^{*})<m(n-m)$,

Norm$(\frac{m^{*}(n-m^{*})}{2},\frac{m^{*}(n-m^{*})(n+1)}{12})$ has smaller expected value and smaller variance, and therefore it has smaller critical values.

Thus,

[TABLE]

where $Z,\ Z^{*}$ are random variables of the Normal Laplace distribution generated by $m$ and $m^{*}$, respectively, and $z^{**}$ is the critical value for the distribution $Z^{*}$, respectively.

Therefore, it is acceptable to use $m^{*}$ to generate the reference distribution (with $1-\delta$ probability).

In conclusion, we can generate the reference distribution using

[TABLE]

Doing so causes the type I error rate to decrease, which is acceptable, even if it means a decrease in power.

∎

Proof of Theorem 5.1

Theorem**.**

Algorithm $\sf{\widetilde{WP}_{\textup{stat}}}$ is $\epsilon$-differentially private.

Proof.

It is sufficient to show that the sensitivity of $\sf{WP}_{\textup{stat}}$ is bounded by $2n$. The privacy of the algorithm then follows directly from Theorem 2.8.

Let x and $\textbf{x}\mathbf{{}^{\prime}}$ be neighbouring databases. Let row $j$ be the row that differs between x and $\textbf{x}\mathbf{{}^{\prime}}$. Let $d_{j}$ and $d^{\prime}_{j}$ indicate the difference between the data values in row $j$ in x and $\textbf{x}\mathbf{{}^{\prime}}$, respectively. Similarly, let $r_{j}$ and $r^{\prime}_{j}$ indicate the rank of row $j$ in x and $\textbf{x}\mathbf{{}^{\prime}}$. We assume without loss of generality that $d^{\prime}_{j}>d_{j}$.

We divide the rows of x (or $\textbf{x}\mathbf{{}^{\prime}}$) into five regions, $A_{1}$ through $A_{5}$. $A_{1}$ is the set of rows $i$ with $d_{i}<d_{j}$. $A_{2}$ is the set with $d_{i}=d_{j}$. $A_{3}$ has $d_{i}$ values between $d_{j}$ and $d^{\prime}_{j}$. $A_{4}$ has $d_{i}$ values equal to $d^{\prime}_{j}$. $A_{5}$ has values greater than $d^{\prime}_{j}$ (though we don’t need to use $A_{5}$ in the proof). For convenience, we use $a_{i}$ to represent the size of the set $A_{i}$ in x.

As shown in Figure 15, x and $\textbf{x}\mathbf{{}^{\prime}}$ differ only in that $d_{j}$ is “moved” from region $A_{2}$ to region $A_{4}$. Let $w$ and $w^{\prime}$ be test statistics produced from databases x and $\textbf{x}\mathbf{{}^{\prime}}$, respectively.

Recall that $w=\sum_{i}s_{i}r_{i}$. Let $w_{i}=s_{i}r_{i}$ be the contribution of row $i$ to this sum. We then want to bound $\Delta w=|w-w^{\prime}|$. We similarly set $\Delta w_{i}$ to be $|s_{i}r_{i}-s^{\prime}_{i}r^{\prime}_{i}|$. Note that for $i$ in $A_{1}$ or $A_{5}$ we have $\Delta w_{i}=0$.

Next, we consider $\Delta w_{j}$ the change in the term corresponding to the row where the data changes. As row $j$ is tied with all other rows in region $A_{2}$, $r_{j}=\frac{(a_{1}+1)+(a_{1}+a_{2})}{2}=\frac{2a_{1}+a_{2}+1}{2}$ and

$r^{\prime}_{j}=\frac{(a_{1}+a_{2}+a_{3})+(a_{1}+a_{2}+a_{3}+a_{4})}{2}=\frac{2a_{1}+2a_{2}+2a_{3}+a_{4}}{2}$. In the worst case, $s_{j}$ changes when the move occurs, so

[TABLE]

Now we consider $\Delta w_{i}$ where $i\in A_{2}$ (excluding row $j$). As there is one fewer row tied, each of their ranks (excluding row $j$) is changed from $\frac{2a_{1}+a_{2}+1}{2}$ to $\frac{2a_{1}+a_{2}}{2}$, which means that each row is changed by $\frac{1}{2}$. As there are $a_{2}-1$ rows in this region, the contribution to $\Delta w$ from this set of rows is bounded by $\frac{a_{2}-1}{2}$.

Similarly, each of the row ranks (excluding row $j$) in region $A_{4}$ is changed from $\frac{2a_{1}+2a_{2}+2a_{3}+a_{4}+1}{2}$ to $\frac{2a_{1}+2a_{2}+2a_{3}+a_{4}}{2}$ as one more row is tied with rows in $A_{4}$. As there are $a_{4}$ rows, $\Delta w$ from the rows in region $A_{4}$ is bounded by $\frac{a_{4}}{2}$.

Now we consider region $A_{3}$. As $d_{j}$ has moved from before this region to after this region, each of the row ranks is changed by $1$. Therefore the contribution to $\Delta w$ from the rows in region $A_{3}$ is bounded by $a_{3}$.

It follows that

[TABLE]

Having bounded the sensitivity, the proof is complete.

∎

Proof of Theorems 5.3 and Theorem 5.4

Theorem**.**

The sensitivity of $\bar{x}$ is $\frac{2}{n}$ .

Proof.

Consider two neighboring databases, $D$ and $D^{\prime}$, of size $n$. Without loss of generality suppose that the observation that differs is $x_{n}$. Then,

[TABLE]

∎

Theorem**.**

The sensitivity of $s^{2}$ is $\frac{5}{n-1}$.

Proof.

Consider two neighboring databases, $D$ and $D^{\prime}$, of size $n$. Without loss of generality suppose that the observation that differs is $x_{n}$. We first simplify $s^{2}$ for sensitivity analysis.

[TABLE]

Then we consider how much $s^{2}$ is differed with one row $x_{n}$ differed.

[TABLE]

By the triangle inequality,

[TABLE]

By Theorem 5.3, $|\bar{x}-\bar{x^{\prime}}|\leq\frac{2}{n}$. It follows that

[TABLE]

∎

Appendix B Supplementary Results

Many Groups

Application to Real-World Data

The Global Findex database maintains survey data from a large number of countries and years about financial inclusion around the world [41]. We will use our differentially private Kruskal Wallis tests to examine whether income quintiles statistically significantly differ by age in the United States.

In Figure 16, we sample a large number of subsets of several different sizes in order to quantify how likely our differentially private variant of the Kruskal-Wallis would be to detect the effect in this dataset as compared to the public version. At an $\epsilon$ of $1$, the absolute value test variant follows extremely closely behind the public version. Even at an $\epsilon$ an order of magnitude smaller, the absolute value test outperforms the standard version. Because these values are randomly sampled, we can also learn from this figure that the test is robust to unequal sample sizes.

Dealing with ties

We now examine the effect of our tiebreaking procedure (randomly ordering all tied values) on the power of the test. We subtract the Gini coefficient [16], a measure of the statistical spread of data, from 1, in order to quantify the proportion of ties per trial of the test procedure.

Varying effect size

In Figure 18, we examine the use case of large sample sizes attempting to detect small effect sizes, as is increasingly common in the scientific literature [4]. We find that, at large sample sizes, the test at $\epsilon=1$ needs only marginally more data than the public test to achieve the same power.

Uniformity of p-values (cont.)

Theorem A.5 shows that the expected value of the statistic is maximized when the groups are exactly equal in size. Thus, in place of the true group sizes, algorithm $\sf{\widetilde{KWabs}_{\textup{p}}}$ assumes that the group sizes are of size $\lfloor\frac{n}{g}\rfloor$ when simulating the reference distribution (in order to maintain privacy.) One may wonder, then, if the type I error rate of our test increases when sample sizes are unequal. We thus fix the total sample size and vary the proportion of the total sample size $n$ that the maximum group sample size $n$ takes up. The results are shown in Figure 19.

As displayed in Figure 19, the type I error rate of our test is consistently below alpha. Further, the power loss due to unequal sample sizes is minimal, even with drastically different group sample sizes.

Further Comparisons to Previous Work

We continue our comparisons to Swanberg et. al.’s test introduced in Section 3 at varying parameterizations in order to better understand the relationship between their test and $\sf{\widetilde{KWabs}_{\textup{p}}}$.

Due to the rank-based nature of $\sf{\widetilde{KWabs}_{\textup{p}}}$, the public nonparametric test statistic (before the addition of noise) does not change once all of the ranks of one group are on one side of the distribution. Thus, we take a particular interest in sampling distributions with an effect size well beyond two standard deviations (the maximum studied in the main body) since the parametric statistic ought to perform better in such situations. We find, however, that $\sf{\widetilde{KWabs}_{\textup{p}}}$ persists in outperforming Swanberg et. al.’s test at the parameterizations chosen to reflect such a setting in Figure 20.

We also consider settings where effect sizes are notably smaller than those studied in the main body in Figure 21.

Two Groups

Application to Real-World Data

In 2017, Andersen et al. conducted a study on patient’s and community health volunteer’s understanding of eligibility for a legal medical abortion in Nepal [1]. The dataset collected for the study was released on the condition that users of the data not seek to deanonymize any of the study participants. We demonstrate that our test reaches a meaningful level of statistical power for a variety of parameters—and could have done so with smaller sample sizes than that taken in this study—and thus a differentially-private query interface would have been a sufficient tool for the release of this data while greatly decreasing the risk of reidentification.

Of the many variables in this dataset, we focus on two; the first is simply a categorical variable indicating whether or not a medical abortion was granted, and the second is the number of days since the patient’s last period. One may wonder if, as the number of days since the patients last period increases, their likelihood of being granted a medical abortion changes. This question, naturally, can be answered by the Mann-Whitney U test. We have 472 cases where medical abortions were not granted (with an average number of days since last period of around 72), and 2620 cases where medical abortions were granted (with an average number of days since last period of around 49). The means of the two groups differ by slightly less than 1.5 standard deviations (where the number of days since last period is greater for subjects not granted a medical abortion).161616These values come from a slightly filtered and tidied version of the original dataset—our source code can be found at: github.com/simonpcouch/non-pm-dpht

In Figure 22 we sample a varying number of observations from the groups and evaluate the power of both our and the public test. The far right of the figure shows the power using almost all of the provided data (while still maintaining equal group sizes). For $\epsilon$ $=$ $1$, 100% power is achieved well before all data is used, while that for $\epsilon$ $=$ $.1$ is more likely than not to detect the effect by the time all of the data is used.

Uniformity of p-values (cont.)

In addition to the examination of uniformity of p-values when group sizes are equal, we also examine the effect of unequal sample sizes in Figure 23 and find that as sample sizes become more unequal, our critical values are increasingly conservative.

While our critical values do not exceed $\alpha$ in any case, they are much too conservative when total sample size is too small or group sample sizes are largely different.

Naturally, one might wonder whether we can assume that group sizes are equal when we generate our reference distribution, as we do with our Kruskal-Wallis, so that we can allot of our privacy budget to estimating the test statistic. In Figure 24, we address this question with a quantile-quantile plot where we fix total sample size and use the same reference distribution (assuming equal group sizes) on null distributions generated with unequal group sizes. We find that, even with $n$ $=$ $1000$ and $\epsilon_{tot}=1$, small differences in group sizes cause egregious increases in the type I error rate. We conclude, as a result, that our allotment of some portion of the privacy budget to estimating group sizes is essential.

Varying effect size

We examine the power of our test at varying effect sizes, as shown in Figure 25. Two features of this graph display notable characteristics of our test. Initially, note how minimal the difference is between the public test and our test at $\epsilon_{tot}=1$; at large sample sizes (in this case, $n=1500$), the power of the public test and that of our test is almost indistinguishable. Further, the $\epsilon_{tot}=.06$ line shows that, once the effect size reaches a difference in $\mu$ of roughly 3$\sigma$, the power of the test seems to level off before reaching $1$. This is due to the rank-based nature of the Mann-Whitney; once all of the ranks of one group are above or below those of the other, increasing effect size has no consequence. Thus, at certain choices of $\epsilon_{tot}$ and $n$, there is effectively an upper bound on the power test that is below $1$, regardless of effect size.

The effect of unequal group sizes

In Figure 26, we quantify the effect of unequal group sizes on the power of the test. In the public setting, power reduction only occurs when the sample sizes are meaningfully different. In the private setting, however, when random noise is added to $n_{1}$ and $n_{2}$ for the simulation of the reference distribution (and thus critical value computation), smaller total sample sizes often result in substantial differences in size of $m^{*}$ and $n-m^{*}$.

Comparative effects of unequal group sizes

We continue the discussion given in Section 4.3 on comparing $\sf{\widetilde{MW}_{\textup{p}}}$ to $\sf{\widetilde{KWabs}_{\textup{p}}}$ in the 2-group use case, this time quantifying the comparative losses in power to the two tests when group sizes are unequal. The $\sf{\widetilde{MW}_{\textup{p}}}$ test does not assume equal sizes, and must rather allot some of the total privacy budget to estimating one of the group sizes $m$ since the Type I error rate increases when equal group sizes are assumed when generating the reference distribution. However, as we have previously shown, it is acceptable for Type I error for $\sf{\widetilde{KWabs}_{\textup{p}}}$ to assume equal group sizes, though this means increases in the Type II error rate (i.e. losses in power) when group sizes are highly unequal. Thus, it is natural to ask whether the $\sf{\widetilde{MW}_{\textup{p}}}$ test becomes more powerful when group sizes are very different since some of the privacy budget is allotted to estimating $m$, and thus, the reference distribution is more precisely calibrated to reduce Type II error rate. Our findings in this setting are summarized in Figures 27 and 28, where we find that $\sf{\widetilde{KWabs}_{\textup{p}}}$ persists in achieving higher power than $\sf{\widetilde{MW}_{\textup{p}}}$ even under such parameterizations.

Known group size special case

As noted in 4, when group sizes are known publicly, the entirety of the privacy budget can be allotted to estimating the test statistic $U$ in the $\sf{\widetilde{MW}_{\textup{p}}}$ algorithm since $m$ need not be estimated. This special use case increases the precision of $\sf{\widetilde{MW}_{\textup{p}}}$ enough to cause the test to reach higher powers than $\sf{\widetilde{KWabs}_{\textup{p}}}$ at equivalent parameterizations, as shown in Figure 29.

Allotment of Privacy Budget

This section serves to justify our choice of the optimum proportion of $\epsilon_{tot}$ to allot to estimating $m$ and the test statistic $\sf{MW_{\textup{stat}}}$.

Initially, we fix all parameters other than the proportion of $\epsilon$ to $m$ and total sample size $n$ and calculate power across many values.

As shown in Figure 30, to maximize power, the optimum proportion of epsilon to allot to estimating $m$ is near $.65$.

Wilcoxon Signed-Rank

Dealing with Ties

In Figure 31, 30% of rows have $d_{i}=0$ and the other 70% are distributed as before, with the two data points sampled from normal distributions with means one standard deviation apart.

Figure 10 uses a continuous distribution for the testing data. To show that our test handles ties more effectively than the TC test, we again introduce ties into the data, this time at varying proportion, and measure power. In particular, Figures 32 and 33 compare our algorithm to the high utility and high privacy variants of the TC test, respectively. Here we first choose the number of rows with $d_{i}=0$ and then sample the remaining data points as before. Of course, as the number of rows showing no difference increases, the power of all tests decreases, but we see that in all cases our test retains power longer.

Varying Effect Size

We also examine the power of our test at varying effect sizes. As shown in Figure 34, at large sample sizes $n$, in this case $2500$, there is essentially no difference between the minimum effect detectable in the private and public setting. This is because the random variation in the sample overwhelms the relatively small random noise being added for privacy. We also note, while not shown in the figure, that for small enough choices of $n$ and $\epsilon$ no effect, no matter how large, can be detected. (Once all $d_{i}$ values are positive, increasing the effect size further has no effect.)

Application to Real-World Data

We now demonstrate the use of our Wilcoxon algorithm on real-world data. We use a database of New York City tax ride information released in 2014. The database contains information on every Yellow Taxi ride in New York City in 2013, and its release resulted in high-profile de-anonymization attacks [38]. Our main finding is that (at least for some natural analyses that we attempt) a differentially private query interface would have been sufficient, and the release of the data set was unnecessary. We also again compare our statistical power to that of the TC tests and find it superior.171717This data set is the same one Task and Clifton originally use for evaluating their test.

This dataset contains several variables of interest for every Yellow Taxi ride in New York City in 2013, of which the following will be most useful:

• •

Hack License: a unique identifier for every taxi driver in the city

• •

Number of Passengers: how many people rode together in the taxi

• •

Trip Time: the duration of the ride in seconds

• •

Trip Distance: the total distance traveled during the ride

We subsetted this dataset, initially, to include rides occuring on January 1st and 2nd of 2013. Then, we only kept rides given by a driver who drove on both the 1st and the 2nd. In one data set, $u_{i}$ and $v_{i}$ were the average trip time on the 1st and 2nd, respectively. The Wilcoxon test can then be used to test whether trip time varied between the two days. We calculated two other data sets similarly for trip distance and number of passengers.

The resulting data sets have 17,066 entries. We sampled (with replacement) $10^{5}$ different smaller datasets of size $n=400$. Finally, we ran each algorithm on each of these resulting data sets and report the proportion in which a significant result (at $\alpha=.05$) was found. The results are summarized in Figure 35.

The power of our test follows the public test closely, and we see that for a data set as small as 400, we can achieve results with a private query that are almost as useful as a full public release of the data. The TC algorithms do not achieve this goal, though the high utility variant is still meaningfully useful. Of course, there will be choices of $n$ and $\epsilon$ for which the gap between our power and the power of the public test is quite large, but our point here is to argue that for even reasonably small data sets a private query interface can be sufficient for many important tasks.

Power Comparison at $\epsilon=.1$

We compare the power of our next algorithm to the previous best at a smaller choice of $\epsilon$.

Critical Value Tables

We present critical values for several choices of $\epsilon$ and sample size.

T-Test

Allotment of Privacy Budget

This section serves to briefly justify our choice of the optimum proportion of the privacy budget to allot to estimating $\widehat{\bar{x}}$ and $\widehat{s^{2}}$.

As evidenced in Figure 37, unlike $\sf{\widetilde{MW}_{\textup{p}}}$, the optimum proportion of $\epsilon$ to allot to either statistic is not obvious; certain proportions offer distinct advantages depending on parameterizations. To most objectively choose our allotment of the privacy budget, we refer to the common 80% power rule used in determining sample size requirements in biological studies and consider the $\epsilon$ allotment that reaches 80% power most quickly to be optimum. Resultantly, we choose to allot equal proportions of the total privacy budget to estimating $\widehat{\bar{x}}$ and $\widehat{s^{2}}$.