F-BLEAU: Fast Black-box Leakage Estimation

TL;DR

This paper introduces a machine learning-based method for efficiently estimating information leakage in black-box systems, especially those with large or continuous output spaces, improving over traditional frequency-based approaches.

Contribution

It proposes using universally consistent ML rules, particularly nearest neighbor methods, to accurately estimate leakage with fewer queries and applicability to continuous outputs.

Findings

ML-based methods reduce the number of queries needed

Nearest neighbor rules handle continuous output systems

Compared favorably with state-of-the-art leakage estimation tools

Abstract

We consider the problem of measuring how much a system reveals about its secret inputs. We work under the black-box setting: we assume no prior knowledge of the system's internals, and we run the system for choices of secrets and measure its leakage from the respective outputs. Our goal is to estimate the Bayes risk, from which one can derive some of the most popular leakage measures (e.g., min-entropy, additive, and multiplicative leakage). The state-of-the-art method for estimating these leakage measures is the frequentist paradigm, which approximates the system's internals by looking at the frequencies of its inputs and outputs. Unfortunately, this does not scale for systems with large output spaces, where it would require too many input-output examples. Consequently, it also cannot be applied to systems with continuous outputs (e.g., time side channels, network traffic). In this paper, we exploit an analogy between Machine Learning (ML) and black-box leakage estimation to show that the Bayes risk of a system can be estimated by using a class of ML methods: the universally consistent learning rules; these rules can exploit patterns in the input-output examples to improve the estimates' convergence, while retaining formal optimality guarantees. We focus on a set of them, the nearest neighbor rules; we show that they significantly reduce the number of black-box queries required for a precise estimation whenever nearby outputs tend to be produced by the same secret; furthermore, some of them can tackle systems with continuous outputs. We illustrate the applicability of these techniques on both synthetic and real-world data, and we compare them with the state-of-the-art tool, leakiEst, which is based on the frequentist approach.

Figures

Tables

Table 1. TABLE I: Number of examples required for convergence of the estimates. “X” means an estimate did not converge.

System	Dataset	frequentist	NN	$k_n$-NN
Random	100 secrets, 100 obs.	10 070	10 070	10 070
Geometric ($\nu =0.1$)	100 secrets, 10K obs.	35 016	333	458
Geometric ($\nu =0.02$)	100 secrets, 10K obs.	152 904	152 698	68 058
Geometric ($\nu =2$)	10K secrets, 1K obs.	95 500	94 204	107 707
Multimodal Geometric ($\nu =0.1$)	100 secrets, 10K obs.	44 715	568	754
Spiky (contrived example)	2 secrets, 10K obs.	22 908	29 863	62 325
Planar Geometric $\nu =2$	Gowalla checkins in San Francisco area	X	X	19 948
Laplacian $\nu =2$	"	N/A	X	19 961
Blahut-Arimoto $\nu =2$	"	1 285	1 170	1 343

Table 2. TABLE II: Symbols table.

Symbol	Description
$s\in 𝕊$	A secret
$o\in 𝕆$	An object/black-box output
$(s,o)\in 𝕊\times 𝕆$	An example
$(\pi ,{𝒞}_{s,o})$	A system, given a set of priors $\pi$ and channel matrix $𝒞$
$\mu$	Distribution induced by a system on $𝕊\times 𝕆$
$f:𝕊\mapsto 𝕆$	A classifier
$\ell$	Loss function w.r.t. which we evaluate a classifier
$R^f$	The expected misclassification error of a classifier $f$
$R^{\ast }$	Bayes risk

Table 3. TABLE III: Estimates’ guarantees as n → ∞ → 𝑛 n\rightarrow\infty

Method	Guarantee	Space $𝕆$
frequentist	$\to R^{\ast }$	finite
NN	$\to R^{\ast }$	finite
$k_n$-NN	$\to R^{\ast }$	infinite, $(d,𝕆)$ separable
NN Bound+	$\le R^{\ast }$	infinite, $(d,𝕆)$ separable

Table 4. TABLE IV: Synthetic systems.

Name	Privacy parameter	$\left|𝕊\right|$	$\left|𝕆\right|$	$R^{\ast }$
Geometric	1.0	100	10K	$\sim 0$
Geometric	0.1	100	10K	0.007
Geometric	0.01	100	10K	0.600
Geometric	0.2	100	1K	0.364
Geometric	0.02	100	10K	0.364
Geometric	0.002	100	100K	0.364
Geometric	2	100K	100K	0.238
Geometric	2	100K	10K	0.924
Multimodal	1.0	100	10K	0.450
Multimodal	0.1	100	10K	0.456
Multimodal	0.01	100	10K	0.797
Spiky	N/A	2	10K	0
Random	N/A	100	100	0.979

Table 5. TABLE V: Convergence of the estimates when varying ν 𝜈 \nu , fixed | 𝕊 | × | 𝕆 | = 100 × 10 ​ K 𝕊 𝕆 100 10 𝐾 |\mathbb{S}|\times|\mathbb{O}|=100\times 10K

				$k_n$-NN
System	$\delta$	Freq.	NN	${\log }_{10}$	$\log$
Geometric $\nu$ = 1.0	0.1	1 994	267	396	679
	0.05	4 216	325	458	781
	0.01	19 828	425	633	899
	0.005	38 621	439	698	904
Geometric $\nu$ = 0.1	0.1	18 110	269	396	673
	0.05	35 016	333	458	768
	0.01	127 206	439	633	899
	0.005	211 742	4 844	698	904
Geometric $\nu$ = 0.01	0.1	105 453	103 357	99 852	34 243
	0.05	205 824	205 266	205 263	199 604

Table 6. Figure 11: Convergence for a Uniform system of size 100 × 100 100 100 100\times 100 .

				$k_n$-NN
System	$\delta$	Freq.	NN	${\log }_{10}$	$\log$
Multimodal $\nu$ = 1.0	0.1	3 008	369	478	897
	0.05	5 938	495	754	1 267
	0.01	26 634	765	1 166	1 487
	0.005	52 081	765	1 166	1 487
Multimodal $\nu$ = 0.1	0.1	24 453	398	554	821
	0.05	44 715	568	754	1 175
	0.01	149 244	4 842	1 166	1 487
	0.005	226 947	79 712	1 166	1 487
Multimodal $\nu$ = 0.01	0.1	27 489	753	900	381
	0.05	103 374	101 664	92 181	31 452

Table 7. TABLE XIV: True Bayes risk and utility for Gowalla dataset defended using various location privacy mechanisms.

Mechanism	$\nu$	$R^{\ast }$	Utility
Blahut-Arimoto	2	0.760	334.611
	4	0.571	160.839
	8	0.428	96.2724
Geometric	2	0.657	288.372
	4	0.456	144.233
	8	0.308	96.0195
Laplacian	2	0.657	288.66
	4	0.456	144.232
	8	0.308	96.212

Table 8. TABLE XV: Number of unique secrets and observations for the time side channel to finite field exponentiation.

Operands’ size	$\left|𝕊\right|$	$\left|𝕆\right|$
4 bits	$2^4$	34
6 bits	$2^6$	123
8 bits	$2^8$	233
10 bits	$2^{10}$	371
12 bits	$2^{12}$	541

Equations

$$R^{\ast }:=1-\sum _{o\in \mathbb{O}}\underset{s\in \mathbb{S}}{\max }{\mathcal{C}}_{s,o}\pi (s).$$

$$R^{\pi }:=1-\underset{s\in \mathbb{S}}{\max }\pi (s).$$

$$R^f=1-\sum _{o\in \mathbb{O}}{\mathcal{C}}_{f(o),o}\pi (f(o))$$

$$P(o|s)\approx {\hat{\mathcal{C}}}_{s,o}:=\frac{|i:o_i=o,s_i=s|}{|i:s_i=s|}.$$

$$R^{Freq}=1-\sum _o{C}_{f^{Freq}(o),o}\pi (f^{Freq}(o))$$

$$f^{\mathit{Freq}}(o)=\begin{cases}\operatorname*{argmax}_{s}(\hat{\mathcal{C}}_{s,o}\,\hat{\pi}(s))&\mbox{if $o$ in training data}\\ \operatorname*{argmax}_{s}\hat{\pi}(s)&\mbox{otherwise}\,,\end{cases}$$

$${\mathbb{E}}_1(\ell \mid \mu ,n)={\mathbb{E}}_2(\ell \mid \mu ,n).$$

$$NN(o)=s_{h(o)}$$

$$h(o)=\underset{i\in I_{min}(o)}{\mathrm{argmax}}|\{j\in I_{min}(o)\mid s_j=s_i\}|.$$

$$R_n^{NN}\to R^{\ast }.$$

$$k\text{mbox}-NN(o)=s_{h(o)}$$

$$h(o)=\underset{i=1,...,k}{\mathrm{argmax}}|\{j\in I_{min}(o)\mid s_{(j)}=s_{(i)}\}|.$$

$$d(o,o_{(k^{\prime })})=...=d(o,o_{(k)})=...=d(o,o_{(k^{\prime \prime })}),$$

$$s_{(1)},s_{(2)},...,s_{(k^{\prime }-1)},\hat{s},\hat{s}...,\hat{s}.$$

$$\frac{\left|R_n^f-R^{\ast }\right|}{R^{\ast }}<\delta .$$

$$\left|R_n^f-R^{\ast }\right|<\delta .$$

$${\mathcal{C}}_{s,o}=P(o\mid s)=\lambda \exp \left(-\nu \mid g(s)-o\mid \right),$$

$$\mathrm{ME}:=-{\log }_2(1-R^{\pi })+{\log }_2(1-R^{\ast }).$$

$$\frac{|\mathbb{S}|-1}{|\mathbb{S}|}\left(1-\sqrt{1-\frac{|\mathbb{S}|}{|\mathbb{S}|-1}{R}^{NN}}\right)\le R^{\ast }.$$

$${\mathcal{C}}_{s,o}=P(o\mid s)=\lambda \exp \left(-\nu \mid g(s)-o\mid \right),$$

$${\Delta }_g=\underset{s_1\sim s_2\in \mathbb{S}}{\max }(g(s_1)-g(s_2)),$$

$$\lambda=\begin{cases}e^{\nu}/(e^{\nu}+1)\quad&\mbox{if $o=1$ or $o=w^{\prime}$}\\ (e^{\nu}-1)/(e^{\nu}+1)\quad&\mbox{otherwise}\,,\end{cases}$$

$${\mathcal{C}}_{s,o}^{\prime }={\mathcal{C}}_{s^{\prime },o}{s}^{\prime }=s\mathrm{mod}|O|,$$

$${\mathcal{C}}_{s,o}^M=w_1{\mathcal{C}}_{s,o}+w_2{\mathcal{C}}_{s+2\sigma ,o}.$$

$${\mathcal{C}}_{s,o}=\left[\begin{array}{ccccc}\text{nicefrac}2q& 0& \text{nicefrac}2q& \dots & 0\\ 0& \text{nicefrac}2q& 0& \dots & \text{nicefrac}2q\end{array}\right].$$

$$r_n(o)=$$

$$=$$

$$+(1-\underset{s\in \mathbb{S}}{\max }\hat{\pi }(s))P(\neg A_n(o))\approx$$

$$\approx$$

$$+(1-\underset{s\in \mathbb{S}}{\max }\pi (s))P(\neg A_n(o))$$

$$R_n^{Freq}=$$

$$=$$

$$+\sum _{o\in \mathbb{O}}P(o)(1-\underset{s\in \mathbb{S}}{\max }\pi (s))P(\neg A_n(o))=$$

Full text

The state-of-the-art method for estimating these leakage measures is the frequentist paradigm, which approximates the system’s internals by looking at the frequencies of its inputs and outputs. Unfortunately, this does not scale for systems with large output spaces, where it would require too many input-output examples. Consequently, it also cannot be applied to systems with continuous outputs (e.g., time side channels, network traffic).

In this paper, we exploit an analogy between Machine Learning (ML) and black-box leakage estimation to show that the Bayes risk of a system can be estimated by using a class of ML methods: the universally consistent learning rules; these rules can exploit patterns in the input-output examples to improve the estimates’ convergence, while retaining formal optimality guarantees. We focus on a set of them, the nearest neighbor rules; we show that they significantly reduce the number of black-box queries required for a precise estimation whenever nearby outputs tend to be produced by the same secret; furthermore, some of them can tackle systems with continuous outputs. We illustrate the applicability of these techniques on both synthetic and real-world data, and we compare them with the state-of-the-art tool, leakiEst, which is based on the frequentist approach.

I Introduction

Measuring the information leakage of a system is one of the founding pillars of security. From side-channels to biases in random number generators, quantifying how much information a system leaks about its secret inputs is crucial for preventing adversaries from exploiting it; this has been the focus of intensive research efforts in the areas of privacy and of quantitative information flow (QIF). Most approaches in the literature are based on the white-box approach, which consists in calculating analytically the channel matrix of the system, constituted by the conditional probabilities of the outputs given the secrets, and then computing the desired leakage measures (for instance, mutual information [1], min-entropy leakage [2], or $g$-leakage [3]). However, while one typically has white-box access to the system they want to secure, determining a system’s leakage analytically is often impractical, due to the size or complexity of its internals, or to the presence of unknown factors. These obstacles led to investigate methods for measuring a system’s leakage in a black-box manner.

Until a decade ago, the most popular measure of leakage was Shannon mutual information (MI). However, in his seminal paper [2] Smith showed that MI is not appropriate to represent a realistic attacker, and proposed a notion of leakage based on Rényi min-entropy (ME) instead. Consequently, in this paper we consider the general problem of estimating the Bayes risk of a system, which is the smallest error achievable by an adversary at predicting its secret inputs given the outputs. From the Bayes risk one can derive several leakage measures, including ME and the additive and multiplicative leakage [4]. These measures are considered by the QIF community among the most fundamental notions of leakage.

To the best of our knowledge, the only existing approach for the black-box estimation of the Bayes risk comes from a classical statistical technique, which refer to as the frequentist paradigm. The idea is to run the system repeatedly on chosen secret inputs, and then count the relative frequencies of the secrets and respective outputs so to estimate their joint probability distribution; from this distribution, it is then possible to compute estimates of the desired leakage measure. LeakWatch [5] and leakiEst [6], two well-known tools for black-box leakage estimation, are applications of this principle.

Unfortunately, the frequentist approach does not always scale for real-world problems: as the number of possible input and output values of the channel matrix increases, the number of examples required for this method to converge becomes too large to gather. For example, LeakWatch requires a number of examples that is much larger than the product of the size of input and output space. For the same reason, this method cannot be used for systems with continuous outputs; indeed, it cannot even be formally constructed in such a case.

Our contribution

In this paper, we show that machine learning (ML) methods can provide the necessary scalability to black-box measurements, and yet maintain formal guarantees on their estimates. By observing a fundamental equivalence between ML and black-box leakage estimation, we show that any ML rule from a certain class (the universally consistent rules) can be used to estimate with arbitrary precision the leakage of a system. In particular, we study rules based on the nearest neighbor principle – namely, Nearest Neighbor (NN) and $k_{n}$-NN, which exploit a metric on the output space to achieve a considerably faster convergence than frequentist approaches. In Table I we summarize the number of examples necessary for the estimators to converge, for the various systems considered in the paper. We focus on nearest neighbor methods, among the existing universally consistent rules, because: i) they are simple to reason about, and ii) we can identify the class of systems for which they will excel, which happens whenever the distribution is somehow regular with respect to a metric on the output (e.g., time side channels, traffic analysis, and most mechanisms used for privacy). Moreover, some of these methods can tackle directly systems with continuous output.

We evaluate these estimators on synthetic data, where we know the true distributions and we can determine exactly when the estimates converge. Furthermore, we use them for measuring the leakage in a real dataset of users’ locations, defended with three state-of-the-art mechanisms: two geo-indistinguishability mechanisms (planar geometric and planar Laplacian)[7], and the method by Oya et al. [8], which we refer to as the Blahut-Arimoto mechanism. Crucially, the planar Laplacian is real-valued, which $k_{n}$-NN methods can tackle out-of-the box, but the frequentist method cannot. Results in both synthetic and real-world data show our methods give a strong advantage whenever there is a notion of metric in the output that can be exploited. Finally, we compare our methods with leakiEst on the problem of estimating the leakage of European passports [9, 6], and on the location privacy data.

As a further evidence of their practicality, we use them in Appendix G to measure the leakage of a time side channel in a hardware implementation of finite field exponentiation.

No Free Lunch

A central takeaway of our work is that, while all the estimators we study (including the frequentist approach) are asymptotically optimal in the number of examples, none of them can guarantee on its finite sample performance; indeed, no estimator can. This is a consequence of the No Free Lunch theorem in ML [10], which informally states that all learning rules are equivalent among the possible distributions of data. This rules out the existence of an optimal estimator.

In practice, this means that we should always evaluate several estimators, and select the one that converged faster. Fortunately, our main finding (i.e., any universally consistent ML rule is a leakage estimator) adds a whole new class of estimators, which one can use in practical applications.

We therefore propose a tool, F-BLEAU (Fast Black-box Leakage Estimation AUtomated), which computes nearest neighbor and frequentist estimates, and selects the one converging faster. We release it as Open Source software111https://github.com/gchers/fbleau., and we hope in the future to extend it to support several more estimators based on UC ML rules.

Nearest Neighbor rules

Nearest neighbor rules excel whenever there is a notion of metric on the output space, and the output distributions is “regular” (in the sense that it does not change too abruptly between two neighboring points). We expect this to be the case for several real-world systems, such as: side channels whose output is time, an electromagnetic signal, or power consumption; for traffic analysis on network packets; and for geographic location data. Moreover, most mechanisms used in privacy and QIF use smooth noise distributions. Suitable applications may also come from recent attacks to ML models, such as model inversion [11] and membership inference [12].

Furthermore, we observe that even when there is no metric, or when the output distribution is irregular, (e.g., a system whose internal distribution has been randomly sampled), these rules are equivalent to the frequentist approach. Indeed, the only case we observe when they are misled is when the system is crafted so that the metric contributes against classification (e.g., see “Spiky” example in Table I).

II Related Work

Chatzikokolakis et al. [13] introduced methods for measuring the leakage of a deterministic program in a black-box manner; these methods worked by collecting a large number of inputs and respective outputs, and by estimating the underlying probability distribution accordingly; this is what we refer to as the frequentist paradigm. A fundamental development of their work by Boreale and Paolini [14] showed that, in the absence of significant a priori information about the output distribution, no estimator does better than the exhaustive enumeration of the input domain. In line with this work, section IV will show that, as a consequence of the No Free Lunch theorem in ML, no leakage estimator can claim to converge faster than any other estimator for all distributions.

The best known tools for black-box estimation of leakage measures based on the Bayes risk (e.g., min-entropy) are leakiEst [15, 6] and LeakWatch [5, 16], both based on the frequentist paradigm. The former also allows a zero-leakage test for systems with continuous outputs. In LABEL:sec:comparison we provide a comparison of leakiEst with our proposal.

Cherubin [17] used the guarantees of nearest neighbor learning rules for estimating the information leakage (in terms of the Bayes risk) of defenses against website fingerprinting attacks in a black-box manner.

Shannon mutual information (MI) is the main alternative to the Bayes risk-based notions of leakage in the QIF literature. Although there is a relation between MI and Bayes risk [18], the corresponding models of attackers are very different: the first corresponds to an attacker who can try infinitely many times to guess the secret, while the second has only one try at his disposal [2]. Consequently, MI and Bayes-risk measures, such as ME, can give very different results: Smith [2] shows two programs that have almost the same MI, but one has an ME several orders of magnitude larger than the other one; conversely, there are examples of two programs such that ME is [math] for both, while the MI is [math] in one case and strictly positive (several bits) in the other one.

In the black-box literature, MI is usually computed by using Kernel Density Estimation, which although only guarantees asymptotic optimality under smoothness assumptions on the distributions. On the other hand, the ML literature offered developments in this area: Belghazi et al. [19] proposed an MI lower bound estimator based on deep neural networks, and proved its consistency (i.e., it converges to the true MI value asymptotically). Similarly, other works constructed MI variational lower bounds [20, 21].

III Preliminaries

We define a system, and show that its leakage can be expressed in terms of the Bayes risk. We then introduce ML notions, which we will later use to estimate the Bayes risk.

III-A Notation

We consider a system $(\pi,\mathcal{C}_{s,o})$, that associates a secret input $s$ to an observation (or object) $o$ in a possibly randomized way. The system is defined by a set of prior probabilities $\pi(s):=P(s)$, $s\in\mathbb{S}$, and a channel matrix $\mathcal{C}$ of size $|\mathbb{S}|\times|\mathbb{O}|$, for which $\mathcal{C}_{s,o}:=P(o|s)$ for $s\in\mathbb{S}$ and $o\in\mathbb{O}$. We call $\mathbb{S}\times\mathbb{O}$ the example space. We assume the system does not change over time; for us, $\mathbb{S}$ is finite, and $\mathbb{O}$ is finite unless otherwise stated.

III-B Leakage Measures

The state-of-the-art in QIF is represented by the leakage measures based on $g$-vulnerability, a family whose most representative member is min-vulnerability [2], the complement of the Bayes risk. This paper is concerned with finding tight estimates of the Bayes risk, which can then be used to estimate the appropriate leakage measure.

Bayes risk

The Bayes risk, $R^{*}$, is the error of the optimal (idealized) classifier for the task of predicting a secret $s$ given an observation $o$ output by a system. It is defined with respect to a loss function $\ell:\mathbb{S}\times\mathbb{S}\mapsto\mathbb{R}_{\geq 0}$, where $\ell(s,s^{\prime})$ is the risk of an adversary predicting $s^{\prime}$ for an observation $o$, when its actual secret is $s$. We focus on the 0-1 loss function, $\ell(s,s^{\prime}):=I(s\neq s^{\prime})$, taking value $1$ if $s\neq s^{\prime}$, [math] otherwise. The Bayes risk of a system $(\pi,\mathcal{C}_{s,o})$ is defined as:

[TABLE]

Random guessing

A baseline for evaluating a system is the error committed by an idealized adversary who knows priors but has no access to the channel, and who’s best strategy is to always output the secret with the highest prior. We call the error of this adversary random guessing error:

[TABLE]

III-C Black-box estimation of $R^{*}$

This paper is concerned with estimating the Bayes risk given $n$ examples sampled from the joint distribution $\mu$ on $\mathbb{S}\times\mathbb{O}$ generated by $(\pi,\mathcal{C}_{s,o})$. By running the system $n$ times on secrets $s_{1},\ldots,s_{n}\in\mathbb{S}$, chosen according to $\pi$, we generate a sequence of corresponding outputs $o_{1},\ldots,o_{n}$, thus forming a training set222In line with the ML literature, we call the training or test “set” what is technically a multiset; also, we loosely use the set notation “$\{\}$” for both sets and multisets when the nature of the object is clear from the context. of examples $\{(o_{1},s_{1}),...,(o_{n},s_{n})\}$. From these data, we aim to make an estimate close to the real Bayes risk.

III-D Learning Rules

We introduce ML rules (or, simply, learning rules), which are algorithms for selecting a classifier given a set of training examples. In this paper, we will use the error of some ML rules as an estimator of the Bayes risk.

Let $\mathcal{F}:=\{f\mid f:\mathbb{O}\mapsto\mathbb{S}\}$ be a set of classifiers. A learning rule is a possibly randomized algorithm that, given a training set $\{(o_{1},s_{1}),...,(o_{n},s_{n})\}$, returns a classifier $f\in\mathcal{F}$, with the goal of minimizing the expected loss $\mathbb{E}\,\ell(f(o),s)$ for a new example $(o,s)$ sampled from $\mu$ [22]. In the case of the 0-1 loss function, the expected loss coincides with the expected probability of error (expected error for short), and if $\mu$ is generated by a system $(\pi,\mathcal{C}_{s,o})$, then the expected error of a classifier $f:\mathbb{O}\mapsto\mathbb{S}$ is:

[TABLE]

where $f(o)$ is the secret predicted for object $o$. If $\mathbb{O}$ is infinite (and $\mu$ is continuous) the summation is replaced by an integral.

III-E Frequentist estimate of $R^{*}$

The frequentist paradigm [13] for measuring the leakage of a channel consists in estimating the probabilities $\mathcal{C}_{s,o}$ by counting their frequency in the training data $(o_{1},s_{1}),...,(o_{n},s_{n})$:

[TABLE]

We can obtain the frequentist error from Equation 3:

[TABLE]

where $f^{\mathit{Freq}}$ is the frequentist classifier, namely:

[TABLE]

where $\hat{\pi}$ is estimated from the examples: $\hat{\pi}(s)=\nicefrac{{|i:s_{i}=s|}}{{n}}$.

Consider a finite example space $\mathbb{S}\times\mathbb{O}$. Provided with enough examples, the frequentist approach always converges: clearly, $\hat{\mathcal{C}}\rightarrow\mathcal{C}$ as $n\rightarrow\infty$, because events’ frequencies converge to their probabilities by the Law of Large Numbers.

However, there is a fundamental issue with this approach. Given a training set $\{(o_{1},s_{1}),...,(o_{n},s_{n})\}$, the frequentist classifier can tell something meaningful (i.e., better than random guessing) for an object $o\in\mathbb{O}$, only as long as $o$ appeared in the training set; but, for very large systems (e.g., those with a large object space), the probability of observing an example for each object becomes small, and the frequentist classifier approaches random guessing. We study this matter further in LABEL:subsec:random-system and Appendix E.

IV No Free Lunch In Learning

The frequentist approach performs well only for objects it has seen in the training data; in the next section, we will introduce estimators that aim to provide good predictions even for unseen objects. However, we shall first answer an important question: is there an estimator that is “optimal” for all systems?

A negative answer to this question is given by the so-called “No Free Lunch” (NFL) theorem by Wolpert [10]. The theorem is formulated for the expected loss of a learning rule on unseen objects (i.e., that were not in the training data), which is referred to as the off-training-set (OTS) loss.

Theorem 1** (No Free Lunch).**

Let $A_{1}$ and $A_{2}$ be two learning rules, $\ell$ a cost function, and $\mu$ a distribution on $\mathbb{S}\times\mathbb{O}$. We indicate by $\mathbb{E}_{i}(\ell\mid\mu,n)$ the OTS loss of $A_{i}$ given $\mu$ and $n$, where the expectation is computed over all the possible training sets of size $n$ sampled from $\mu$. Then, if we take the uniform average among all possible distributions $\mu$, we have

[TABLE]

Intuitively, the NFL theorem says that, if all distributions (and, therefore, all channel matrices) are equally likely, then all learning algorithms are equivalent. Remarkably, this holds for any strategy, even if one of the rules is random guessing.

An important implication of this for our purposes is that for every two learning rules $A$ and $B$ there will always exist some system for which rule $A$ converges faster than $B$, and vice versa there will be a system for which $B$ outperforms $A$.

From the practical perspective of black-box security, this demonstrates that we should always test several estimators and select the one that converges faster. Fortunately, the connection between ML and black-box security we highlight in this paper results in the discovery of a whole class of new estimators.

V Machine Learning Estimates of the Bayes Risk

In this section, we define the notion of a universally consistent learning rule, and show that the error of a classifier selected according to such a rule can be used for estimating the Bayes risk. Then, we introduce various universally consistent rules based on the nearest neighbor principle.

Throughout the section, we use interchangeably a system $(\pi,\mathcal{C}_{s,o})$ and its corresponding joint distribution $\mu$ on $\mathbb{S}\times\mathbb{O}$. Note that there is a one-to-one correspondence between them.

V-A Universally Consistent Rules

Consider a distribution $\mu$ and a learning rule $A$ selecting a classifier $f_{n}\in\mathcal{F}$ according to $n$ training examples sampled from $\mu$. Intuitively, as the available training data increases, we would like the expected error of $f_{n}$ for a new example $(o,s)$ sampled from $\mu$ to be minimized (i.e., to get close the Bayes risk). The following definition captures this intuition.

Definition 1** (Consistent Learning Rule).**

Let $\mu$ be a distribution on ${\mathbb{S}\times\mathbb{O}}$ and let $A$ be a learning rule. Let $f_{n}\in\mathcal{F}$ be a classifier selected by $A$ using $n$ training examples sampled from $\mu$. Let $(\pi,\mathcal{C}_{s,o})$ be the system corresponding to $\mu$, and let $R^{f_{n}}$ be the expected error of $f_{n}$, as defined by (3). We say that $A$ is consistent if $R^{f_{n}}\rightarrow R^{*}$ as $n\rightarrow\infty$.

The next definition strengthens this property, by asking the rule to be consistent for all distributions:

Definition 2** (Universally Consistent (UC) Learning Rule).**

A learning rule is universally consistent if it is consistent for any distribution $\mu$ on $\mathbb{S}\times\mathbb{O}$.

By this definition, the expected error of a classifier selected according to a universally consistent rule is also an estimator of the Bayes risk, since it converges to $R^{*}$ as $n\rightarrow\infty$.

In the rest of this section we introduce Bayes risk estimates based on universally consistent nearest neighbor rules; they are summarized in Table III together with their guarantees.

V-B NN estimate

The Nearest Neighbor (NN) is one of the simplest ML classifiers: given a training set and a new object $o$, it predicts the secret of its closest training observation (nearest neighbor). It is defined both for finite and infinite object spaces, although it is UC only in the first case.

We introduce a formulation of NN, which can be seen as an extension of the frequentist approach, that takes into account ties (i.e., neighbors that are equally close to the new object $o$), and which guarantees consistency when $\mathbb{O}$ is finite.

Consider a training set $\{(o_{1},s_{1}),...,(o_{n},s_{n})\}$, an object $o$, and a distance metric $d:\mathbb{O}\times\mathbb{O}\mapsto\mathbb{R}_{\geq 0}$. The NN classifier predicts a secret for $o$ by taking a majority vote over the set of secrets whose objects have the smallest distance to $o$. Formally, let $I_{\mathit{min}}(o)=\{i\mid d(o,o_{i})=\min_{j=1...n}d(o,o_{j})\}$ and define:

[TABLE]

where

[TABLE]

We show that NN is universally consistent for finite $\mathbb{S}\times\mathbb{O}$.

Theorem 2** (Universal consistency of NN).**

Consider a distribution on $\mathbb{S}\times\mathbb{O}$, where $\mathbb{S}$ and $\mathbb{O}$ are finite. Let $R^{\mathit{NN}}_{n}$ be the expected error of the NN classifier for a new observation $o$. As the number of training examples $n\rightarrow\infty$:

[TABLE]

Sketch proof.

For an observation $o$ that appears in the training set, the NN classifier is equivalent to the frequentist approach. For a finite space $\mathbb{S}\times\mathbb{O}$, as $n\rightarrow\infty$, the probability that the training set contains all $o\in\mathbb{O}$ approaches $1$. Thus, the NN rule is asymptotically (in $n$) equivalent to the frequentist approach, which means its error also converges to $R^{*}$. ∎

V-C $k_{n}$-NN estimate

Whilst NN guarantees universal consistency in finite example spaces, this does not hold for infinite $\mathbb{O}$. In this case, we can achieve universal consistency with the k-NN classifier, an extension of NN, for appropriate choices of the parameter $k$.

The k-NN classifier takes a majority vote among the secrets of its neighbors. Breaking ties in the k-NN definition requires more care than with NN. In the literature, this is generally done via strategies that add randomness or arbitrariness to the choice (e.g., if two neighbors have the same distance, select the one with the smallest index in the training data) [23]. We use a novel tie-breaking strategy, which takes into account ties, but gives more importance to the closest neighbors. In early experiments, we observed this strategy had a faster convergence than standard approaches.

Consider a training set $\{(o_{1},s_{1}),...,(o_{n},s_{n})\}$, an object to predict $o$, and some metric $d:\mathbb{O}\times\mathbb{O}\mapsto\mathbb{R}_{\geq 0}$. Let $o_{(i)}$ denote the $i$-th closest object to $o$, and $s_{(i)}$ its respective secret. If ties do not occur after the $k$-th neighbor (i.e., if $d(o,o_{(k)})\neq d(o,o_{(k+1)})$), then k-NN outputs the most frequent among the secrets of the first $k$ neighbors:

[TABLE]

where

[TABLE]

If ties exist after the $k$-th neighbor, that is, for $k^{\prime}\leq k<k^{\prime\prime}$:

[TABLE]

we proceed as follows. Let $\hat{s}$ be the most frequent secret in $\left\{s_{(k^{\prime})},...,s_{(k^{\prime\prime})}\right\}$; k-NN predicts the most frequent secret in the following multiset, truncated at the tail to have size $k$:

[TABLE]

We now define $k_{n}$-NN, a universally consistent learning rule that selects a k-NN classifier for a training set of $n$ examples by choosing $k$ as a function of $n$.

Definition 3** ($k_{n}$-NN rule).**

Given a training set of $n$ examples, the $k_{n}$-NN rule selects a k-NN classifier, where $k$ is chosen such that $k_{n}\rightarrow\infty$ and $k_{n}/n\rightarrow 0$ as $n\rightarrow\infty$.

Stone proved that $k_{n}$-NN is universally consistent [24]:

Theorem 3** (Universal consistency of the $k_{n}$-NN rule).**

Consider a probability distribution $\mu$ on the example space $\mathbb{S}\times\mathbb{O}$, where $\mu$ has a density. Select a distance metric $d$ such that $(d,\mathbb{O})$ is separable333A separable space is a space containing a countable dense subset; e.g., finite spaces and the space of $q$-dimensional vectors $\mathbb{R}^{q}$ with Euclidean metric.. Then the expected error of the $k_{n}$-NN rule converges to $R^{*}$ as $n\rightarrow\infty$.

This holds for any distance metric. In our experiments, we will use the Euclidean distance, and we will evaluate $k_{n}$-NN rules for $k_{n}=\log{n}$ (natural logarithm) and $k_{n}=\log_{10}{n}$.

The ML literature is rich of UC rules and other useful tools for black-box security; we list some of them in Appendix A.

VI Evaluation on Synthetic Data

In this section we evaluate our estimates on several synthetic systems for which the channel matrix is known. For each system, we sample $n$ examples from its distribution, and then compute the estimate on the whole object space as in Equation 3; this is possible because $\mathbb{O}$ is finite. Since for synthetic data we know the real Bayes risk, we can measure how many examples are required for the convergence of each estimate. We do this as follows: let $R^{f}_{n}$ be an estimate of $R^{*}$, trained on a dataset of $n$ examples. We say the estimate $\delta$-converged to $R^{*}$ after $n$ examples if its relative change from $R^{*}$ is smaller than $\delta$:

[TABLE]

While relative change has the advantage of taking into account the magnitude of the compared values, it is not defined when the denominator is [math]; therefore, when $R^{*}\approx 0$ (Table IV), we verify convergence with the absolute change:

[TABLE]

The systems used in our experiments are briefly discussed in this section, and summarized in Table IV; we detail them in Appendix B. A uniform prior is assumed in all cases.

VI-A Geometric systems

We first consider systems generated by adding geometric noise to the secret, one of the typical mechanisms used to implement differential privacy [25]. Their channel matrix has the following form:

[TABLE]

where $\nu$ is a privacy parameter, $\lambda$ a normalization factor, and $g$ a function $\mathbb{S}\mapsto\mathbb{O}$; a detailed description of these systems is given in in Appendix B.

We consider the following three parameters:

• •

the privacy parameter $\nu$,

• •

the ratio $\nicefrac{{|\mathbb{O}|}}{{|\mathbb{S}|}}$, and

• •

the size of the secret space $|\mathbb{S}|$.

We vary each of these parameters one at a time, to isolate their effect on the convergence rate.

VI-A1 Variation of the privacy parameter $\nu$

We fix $|\mathbb{S}|=100$, $|\mathbb{O}|=10$K, and we consider three cases $\nu=1.0$, $\nu=0.1$ and $\nu=0.01$. The results for the estimation of the Bayes risk and the convergence rate are illustrated in Figure 1 and subsubsection VI-A1 respectively. In the table, results are reported for convergence level $\delta\in\{0.1,0.05,0.01,0.005\}$; an “X” means a particular estimate did not converge within 500K examples; a missing row for a certain $\delta$ means no estimate converged.

The results indicate that the nearest neighbor methods have a much faster convergence than the standard frequentist approach, particularly when dealing with large systems. The reason is that geometric systems have a regular behavior with respect to the Euclidean metric, which can be exploited by NN and $k_{n}$-NN to make good predictions for unseen objects.