Differentially Private Kolmogorov-Smirnov-Type Tests

TL;DR

This paper introduces new differentially private nonparametric hypothesis tests based on Kolmogorov-Smirnov and related statistics, which are both statistically valid and require minimal noise, outperforming existing tests especially with small privacy budgets.

Contribution

The paper develops novel DP nonparametric tests using KS, Kuiper, Cramér-von Mises, and Wasserstein statistics with low sensitivity, enabling effective hypothesis testing under privacy constraints.

Findings

Tests have low sensitivity, requiring minimal noise for DP.

Distribution-free null distribution allows easy p-value computation.

Outperform existing DP tests with small privacy budgets or heavy-tailed data.

Abstract

Hypothesis testing is a central problem in statistical analysis, and there is currently a lack of differentially private tests which are both statistically valid and powerful. In this paper, we develop several new differentially private (DP) nonparametric hypothesis tests. Our tests are based on Kolmogorov-Smirnov, Kuiper, Cram\'er-von Mises, and Wasserstein test statistics, which can all be expressed as a pseudo-metric on empirical cumulative distribution functions (ecdfs), and can be used to test hypotheses on goodness-of-fit, two samples, and paired data. We show that these test statistics have low sensitivity, requiring minimal noise to satisfy DP. In particular, we show that the sensitivity of these test statistics can be expressed in terms of the base sensitivity, which is the pseudo-metric distance between the ecdfs of adjacent databases and is easily calculated. The sampling distribution of our test statistics are distribution-free under the null hypothesis, enabling easy computation of $p$-values by Monte Carlo methods. We show that in several settings, especially with small privacy budgets or heavy-tailed data, our new DP tests outperform alternative nonparametric DP tests.