Achieving Privacy in the Adversarial Multi-Armed Bandit

TL;DR

This paper presents a novel approach combining differential privacy mechanisms with the EXP3 algorithm to achieve improved privacy guarantees and regret bounds in adversarial multi-armed bandit problems, supported by empirical validation.

Contribution

It introduces a new method that enhances privacy in adversarial bandits by integrating Laplace and exponential mechanisms with EXP3, improving regret bounds and privacy levels.

Findings

Achieves $ ilde{O}(rac{ oot{2} ext{T} ext{ln T}}{ ext{ε}})$ regret bound with differential privacy.

Improves privacy from linear leakage to $ ilde{O}( oot{ln T})$-DP in EXP3.

Demonstrates empirical results aligning with theoretical improvements.

Abstract

In this paper, we improve the previously best known regret bound to achieve $\epsilon$-differential privacy in oblivious adversarial bandits from $\mathcal{O}{(T^{2/3}/\epsilon)}$ to $\mathcal{O}{(\sqrt{T} \ln T /\epsilon)}$. This is achieved by combining a Laplace Mechanism with EXP3. We show that though EXP3 is already differentially private, it leaks a linear amount of information in $T$. However, we can improve this privacy by relying on its intrinsic exponential mechanism for selecting actions. This allows us to reach $\mathcal{O}{(\sqrt{\ln T})}$-DP, with a regret of $\mathcal{O}{(T^{2/3})}$ that holds against an adaptive adversary, an improvement from the best known of $\mathcal{O}{(T^{3/4})}$. This is done by using an algorithm that run EXP3 in a mini-batch loop. Finally, we run experiments that clearly demonstrate the validity of our theoretical analysis.

Figures

Equations

$$\mathcal{R}=\underset{i=1,\dots K}{\max }\sum _{t=1}^T{g}_{i,t}-\sum _{t=1}^T{g}_{I_t,t}$$

$$\mathop{\mbox{$\mathbb{E}$}}\nolimits\mathcal{R}=\mathop{\mbox{$\mathbb{E}$}}\nolimits\left[\max_{i=1,\ldots K}\sum_{t=1}^{T}g_{i,t}-\sum_{t=1}^{T}g_{I_{t},t}\right]$$

$$p_{i,t}=(1-\gamma )\frac{\exp (\gamma /K{\tilde{G}}_{i,t})}{{\sum }_{i=1}^K\exp (\gamma /K{\tilde{G}}_{i,t})}+\frac{\gamma }{K}$$

$$\displaystyle\mathop{\mbox{$\mathbb{P}$}}\nolimits_{\Lambda}(I_{t}\in S\mid{g}_{1:t-1})\leq\delta+\mathop{\mbox{$\mathbb{P}$}}\nolimits_{\Lambda}(I_{t}\in S\mid{g}^{\prime}_{1:t-1})e^{\epsilon},$$

$$\displaystyle\mathop{\mbox{$\mathbb{P}$}}\nolimits_{\Lambda}(I_{1:t}\in S\mid{g}_{1:t-1})$$

$$\displaystyle\mathop{\mbox{$\mathbb{E}$}}\nolimits R_{{\emph{DP-}\Lambda\emph{-Lap}}}\leq\mathop{\mbox{$\mathbb{E}$}}\nolimits R_{{{\Lambda}}}^{scaled}+2TK\exp(-\epsilon b)+\frac{\sqrt{32T}}{\epsilon}$$

$$\displaystyle\mathop{\mbox{$\mathbb{E}$}}\nolimits R_{{\emph{DP-EXP3-Lap}}}$$

$$+2K+\frac{\sqrt{32T}}{ϵ}$$

$$\min \left\{2T,T\cdot \ln \frac{K(1-\gamma )+\gamma }{\gamma },2(1-\gamma )T+2\sqrt{\frac{2\ln T}{T}}\right\}$$

$$\sqrt{7T\tau K\ln K}+\frac{Tm}{\tau }+\tau$$

Full text

1 Introduction

We consider multi-armed bandit problems in the adversarial setting whereby an agent selects one from a number of alternatives (called arms) at each round and receives a gain that depends on its choice. The agent’s goal is to maximize its total gain over time. There are two main settings for the bandit problem. In the stochastic one, the gains of each arm are generated i.i.d by some unknown probability law. In the adversarial setting, which is the focus of this paper, the gains are generated adversarially. We are interested in finding algorithms with a total gain over $T$ rounds not much smaller than that of an oracle with additional knowledge about the problem. In both settings, algorithms that achieve the optimal (problem-independent) regret bound of $\mathcal{O}(\sqrt{T})$ are known (?; ?; ?; ?; ?; ?; ?).

This problem is a model for many applications where there is a need for trading-off exploration and exploitation. This is so because, whenever we make a choice, we only observe the gain generated by that choice, and not the gains that we could have obtained otherwise. An example is clinical trials, where arms correspond to different treatments or tests, and the goal is to maximize the number of cured patients over time while being uncertain about the effects of treatments. Other problems, such as search engine advertisement and movie recommendations can be formalized similarly (?).

Privacy can be a serious issue in the bandit setting (c.f.  (?; ?; ?; ?)). For example, in clinical trials, we may want to detect and publish results about the best drug without leaking sensitive information, such as the patient’s health condition and genome. Differential privacy (?) formally bounds the amount of information that a third party can learn no matter their power or side information.

Differential privacy has been used before in the stochastic setting (?; ?; ?) where the authors obtain optimal algorithms up to logarithmic factors. In the adversarial setting, (?) adapts an algorithm called Follow The Approximate Leader to make it private and obtain a regret bound of $\mathcal{O}(T^{2/3})$. In this work, we show that a number of simple algorithms can satisfy privacy guarantees, while achieving nearly optimal regret (up to logarithmic factors) that scales naturally with the level of privacy desired.

Our work is also of independent interest for non-private multi-armed bandit algorithms, as there are competitive with the current state of the art against switching-cost adversaries (where we recover the optimal bound). Finally, we provide rigorous empirical results against a variety of adversaries.

The following section gives the main background and notations. Section 3.1 describes meta-algorithms that perturb the gain sequence to achieve privacy, while Section 3.2 explains how to leverage the privacy inherent in the EXP3 algorithm by modifying the way gains are used. Section 4 compares our algorithms with EXP3 in a variety of settings. The full proofs of all our main results are in the full version.

2 Preliminaries

2.1 The Multi-Armed Bandit problem

Formally, a bandit game is defined between an adversary and an agent as follows: there is a set of $K$ arms $\mathcal{A}$, and at each round $t$, the agent plays an arm $I_{t}\in\mathcal{A}$. Given the choice $I_{t}$, the adversary grants the agent a gain $g_{I_{t},t}\in[0,1]$. The agent only observes the gain of arm $I_{t}$, and not that of any other arms. The goal of this agent is to maximize its total gain after $T$ rounds, $\sum_{t=1}^{T}g_{I_{t},t}$. A randomized bandit algorithm $\Lambda:({\mathcal{A}}\times[0,1])^{*}\to\mathscr{D}({\mathcal{A}})$ maps every arm-gain history to a distribution over the next arm to take.

The nature of the adversary, and specifically, how the gains are generated, determines the nature of the game. For the stochastic adversary  (?; ?), the gain obtained at round $t$ is generated i.i.d from a distribution $P_{I_{t}}$. The more general fully oblivious adversary (?) generates the gains independently at round $t$ but not necessarily identically from a distribution $P_{I_{t},t}$. Finally, we have the oblivious adversary (?) whose only constraint is to generate the gain $g_{I_{t},t}$ as a function of the current action $I_{t}$ only, i.e. ignoring previous actions and gains.

While focusing on oblivious adversaries, we discovered that by targeting differential privacy we can also compete against the stronger $m$-bounded memory adaptive adversary (?; ?; ?) who can use up to the last $m$ gains. The oblivious adversary is a special case with $m=0$. Another special case of this adversary is the one with switching costs, who penalises the agent whenever he switches arms, by giving the lowest possible gain of 0 (here $m=1$).

Regret.

Relying on the cumulative gain of an agent to evaluate its performance can be misleading. Indeed, consider the case where an adversary gives a zero gain for all arms at every round. The cumulative gain of the agent would look bad but no other agents could have done better. This is why one compares the gap between the agent’s cumulative gain and the one obtained by some hypothetical agent, called oracle, with additional information or computational power. This gap is called the regret.

There are also variants of the oracle that are considered in the literature. The most common variant is the fixed oracle, which always plays the best fixed arm in hindsight. The regret $\mathcal{R}$ against this oracle is :

[TABLE]

In practice, we either prove a high probability bound on $\mathcal{R}$ or an expected value \mathop{\mbox{\mathbb{E}}}\nolimits\mathcal{R} with:

[TABLE]

where the expectation is taken with respect to the random choices of both the agent and adversary. There are other oracles like the shifting oracle but those are out of scope of this paper.

EXP3.

The Exponential-weight for Exploration and Exploitation (EXP3 (?)) algorithm achieves the optimal bound (up to logarithmic factors) of $\mathcal{O}(\sqrt{TK\ln K})$ for the weak regret (i.e. the expected regret compared to the fixed oracle) against an oblivious adversary. EXP3 simply maintains an estimate $\tilde{G}_{i,t}$ for the cumulative gain of arm $i$ up to round $t$ with $\tilde{G}_{i,t}=\sum_{s=1}^{t}\frac{g_{i,t}}{p_{i,t}}{\mathds{1}}_{I_{t}=i}$ where

[TABLE]

with $\gamma$ a well defined constant.

Finally, EXP3 plays one action randomly according to the probability distribution $p_{t}=\{p_{1,t},\ldots p_{K,t}\}$ with $p_{i,t}$ as defined above.

2.2 Differential Privacy

The following definition (from (?)) specifies what is meant when we called a bandit algorithm differentially private at a single round $t$:

Definition 2.1** (Single round ($\epsilon,\delta)$-differentially private bandit algorithm).**

A randomized bandit algorithm $\Lambda$ is $(\epsilon,\delta)$-differentially private at round $t$, if for all sequence ${g}_{1:t-1}$ and ${g}^{\prime}_{1:t-1}$ that differs in at most one round, we have for any action subset $S\subseteq\mathcal{A}$:

[TABLE]

where \mathop{\mbox{\mathbb{P}}}\nolimits_{\Lambda} denotes the probability distribution specified by the algorithm and $g_{1:t-1}=\{g_{1},\ldots g_{t-1}\}$ with $g_{s}$ the gains of all arms at round $s$. When $\delta=0$, the algorithm is said to be $\epsilon$-differential private.

The $\epsilon$ and $\delta$ parameters quantify the amount of privacy loss. Lower ($\epsilon$,$\delta$) indicate higher privacy and consequently we will also refer to ($\epsilon$,$\delta$) as the privacy loss. Definition 2.1 means that the output of the bandit algorithm at round $t$ is almost insensible to any single change in the gains sequence. This implies that whether or not we remove a single round, replace the gains, the bandit algorithm will still play almost the same action. Assuming the gains at round $t$ are linked to a user private data (for example his cancer status or the advertisement he clicked), the definition preserves the privacy of that user against any third parties looking at the output. This is the case because the choices or the participation of that user would not almost affect the output. Equation (2.2) specifies how much the output is affected by a single user.

We would like Definition 2.1 to hold for all rounds, so as to protect the privacy of all users. If it does for some $(\epsilon,\delta)$, then we say the algorithm has per-round or instantaneous privacy loss $(\epsilon,\delta)$. Such an algorithm also has a cumulative privacy loss of at most $(\epsilon^{\prime},\delta^{\prime})$ with $\epsilon^{\prime}=\epsilon T$ and $\delta^{\prime}=\delta T$ after $T$ steps. Our goal is to design bandit algorithm such that their cumulative privacy loss $(\epsilon^{\prime},\delta^{\prime})$ are as low as possible while achieving simultaneously a very low regret. In practice, we would like $\epsilon^{\prime}$ and the regret to be sub-linear while $\delta^{\prime}$ should be a very small quantity. Definition 2.2 formalizes clearly the meaning of this cumulative privacy loss and for ease of presentation, we will ignore the term ”cumulative” when referring to it.

Definition 2.2** (($\epsilon,\delta)$-differentially private bandit algorithm).**

A randomized bandit algorithm $\Lambda$ is $(\epsilon,\delta)$-differentially private up to round $t$, if for all ${g}_{1:t-1}$ and ${g}^{\prime}_{1:t-1}$ that differs in at most one round, we have for any action subset $S\subseteq\mathcal{A}^{t}$:

[TABLE]

where \mathop{\mbox{\mathbb{P}}}\nolimits_{\Lambda} and ${g}$ are as defined in Definition 2.1.

Most of the time, we will refer to Definition 2.2 and whenever we need to use Definition 2.1, this will be made explicit.

The simplest mechanism to achieve differential privacy for a function is to add Laplace noise of scale proportional to its sensitivity. The sensitivity is the maximum amount by which the value of the function can change if we change a single element in the inputs sequence. For example, if the input is a stream of numbers in $[0,1]$ and the function their sum, we can add Laplace noise of scale $\frac{1}{\epsilon}$ to each number and achieve $\epsilon$-differential privacy with an error of $\mathcal{O}(\sqrt{T}/\epsilon)$ in the sum. However, (?) introduced Hybrid Mechanism, which achieves $\epsilon$-differential privacy with only poly-logarithmic error (with respect to the true sum). The idea is to group the stream of numbers in a binary tree and only add a Laplace noise at the nodes of the tree.

As demonstrated above, the main challenge with differential privacy is thus to trade-off optimally privacy and utility.

Notation.

In this paper, $i$ will be used as an index for an arbitrary arm in $[1,K]$, while $k$ will be used to indicate an optimal arm and $I_{t}$ is the arm played by an agent at round $t$. We use $g_{i,t}$ to indicate the gain of the $i$-th arm at round $t$. $\mathcal{R}_{{{\Lambda}}{}}(T)$ is the regret of the algorithm $\Lambda$ after $T$ rounds. The index and $T$ are dropped when it is clear from the context. Unless otherwise specified, the regret is defined for oblivious adversaries against the fixed oracle. We use ”$x\sim P$” to denote that $x$ is generated from distribution $P$. $\mathop{\mathpzc{Lap}}\nolimits(\lambda)$ is used to denote the Laplace distribution with scale $\lambda$ while $\mathop{\mathpzc{Bern}}\nolimits\left({p}\right)$ denotes the Bernoulli distribution with parameter $p$.

3 Algorithms and Analysis

3.1 $\emph{DP-}\Lambda\emph{-Lap}$: Differential privacy through additional noise

We start by showing that the obvious technique to achieve a given $\epsilon$-differential privacy in adversarial bandits already beat the state-of-the art. The main idea is to use any base bandit algorithm ${\Lambda}$ as input and add a Laplace noise of scale $\frac{1}{\epsilon}$ to each gain before ${\Lambda}$ observes it. This technique gives $\epsilon$-DP differential privacy as the gains are bounded in $[0,1]$ and the noises are added i.i.d at each round.

However, bandits algorithms require bounded gains while the noisy gains are not. The trick is to ignore rounds where the noisy gains fall outside an interval of the form $[-b,b+1]$. We pick the threshold $b$ such that, with high probability, the noisy gains will be inside the interval $[-b,b+1]$. More precisely, $b$ can be chosen such that with high probability, the number of rounds ignored is lower than the upper bound $R_{{{\Lambda}}}$ on the regret of ${\Lambda}$. Given that in the standard bandit problem, the gains are bounded in $[0,1]$, the gains at accepted rounds are rescaled back to $[0,1]$.

Theorem 3.2 shows that all these operations still preserve $\epsilon$-DP while Theorem 3.1 demonstrates that the upper bound on the expected regret of $\emph{DP-}\Lambda\emph{-Lap}$ adds some small additional terms to $R_{{{\Lambda}}}$. To illustrate how small those additional terms are, we instantiate $\emph{DP-}\Lambda\emph{-Lap}$ with the EXP3 algorithm. This leads to a mechanism called DP-EXP3-Lap described in Algorithm 1. With a carefully chosen threshold $b$, corollary 3.1 implies that the additional terms are such that the expected regret of DP-EXP3-Lap is $\mathcal{O}(\sqrt{T}\ln T/\epsilon)$ which is optimal in $T$ up to some logarithmic factors. This result is a significant improvement over the best known bound so far of $\mathcal{O}(T^{2/3}/\epsilon)$ from (?) and solves simultaneously the challenge (whether or not one can get $\epsilon$-DP mechanism with optimal regret) posed by the authors.

Theorem 3.1**.**

If $\emph{DP-}\Lambda\emph{-Lap}$ is run with input a base bandit algorithm ${\Lambda}$, the noisy reward $g^{\prime}_{I_{t},t}$ of the true reward $g_{I_{t},t}$ set to $g^{\prime}_{I_{t},t}=g_{I_{t},t}+{\mathcal{N}}_{I_{t},t}$ with ${\mathcal{N}}_{I_{t},t}\sim\mathop{\mathpzc{Lap}}\nolimits(\frac{1}{\epsilon})$, the acceptance interval set to $[-b,b+1]$ with the scaling of the rewards $g^{\prime}_{I_{t}}$ outside $[0,1]$ done using $g^{\prime}_{I_{t},t}=\frac{g^{\prime}_{I_{t},t}+b}{2b+1}$; then the regret $R_{{\emph{DP-}\Lambda\emph{-Lap}}}$ of $\emph{DP-}\Lambda\emph{-Lap}$ satisfies:

[TABLE]

where $R_{{{\Lambda}}}^{scaled}$ is the upper bound on the regret of ${\Lambda}$ when the rewards are scaled from $[-b,b+1]$ to $[0,1]$

Proof Sketch.

We observed that $\emph{DP-}\Lambda\emph{-Lap}$ is an instance of ${\Lambda}$ run with the noisy rewards $g^{\prime}$ instead of $g$. This means $R_{{{\Lambda}}}^{scaled}$ is an upper bound of the regret $L$ on $g^{\prime}$. Then, we derived a lower bound on $L$ showing how close it is to $R_{{\emph{DP-}\Lambda\emph{-Lap}}}$. This allows us to conclude. ∎

Corollary 3.1**.**

If $\emph{DP-}\Lambda\emph{-Lap}$ is run with EXP3 as its base algorithm and $b=\frac{\ln T}{\epsilon}$, then its expected regret \mathop{\mbox{\mathbb{E}}}\nolimits R_{{\emph{DP-EXP3-Lap}}} satisfies

[TABLE]

Proof.

The proof comes by combining the regret of EXP3 (?) with Theorem 3.1 ∎

Theorem 3.2**.**

$\emph{DP-}\Lambda\emph{-Lap}$* is $\epsilon$-differentially private up to round $T$.*

Proof Sketch.

Combining the privacy of Laplace Mechanism with the parallel composition (?) and post-processing theorems (?) concludes the proof. ∎

3.2 Leveraging the inherent privacy of EXP3

On the differential privacy of EXP3

(?) shows that a variation of EXP3 for the full-information setting (where the agent observes the gain of all arms at any round regardless of what he played) is already differentially private. Their results imply that one can achieve the optimal regret with only a sub-logarithmic privacy loss ($\mathcal{O}(\sqrt{128\log T})$) after $T$ rounds.

We start this section by showing a similar result for EXP3 in Theorem 3.3. Indeed, we show that EXP3 is already differentially private but with a per-round privacy loss of $2$. 111Assuming we want a sub-linear regret. See Theorem 3.3 Our results imply that EXP3 can achieve the optimal regret albeit with a linear privacy loss of $\mathcal{O}(2T)$-DP after $T$ rounds. This is a huge gap compared with the full-information setting and underlines the significance of our result in section 3.1 where we describe a concrete algorithm demonstrating that the optimal regret can be achieved with only a logarithmic privacy loss after $T$ rounds.

Theorem 3.3**.**

The EXP3 algorithm is:

[TABLE]

differentially private up to round $T$.

In practice, we also want EXP3 to have a sub-linear regret. This implies that $\gamma<<1$ and EXP3 is simply $2T$-DP over $T$ rounds.

Proof Sketch.

The first two terms in the theorem come from the observation that EXP3 is a combination of two mechanisms: the Exponential Mechanism (?) and a randomized response. The last term comes from the observation that with probability $\gamma$ we enjoy a perfect [math]-DP. Then, we use Chernoff to bound with high probability the number of times we suffer a non-zero privacy loss. ∎

We will now show that the privacy of EXP3 itself may be improved without any additional noise, and with only a moderate impact on the regret.

On the privacy of a EXP3 wrapper algorithm

The previous paragraph leads to the conclusion that it is impossible to obtain a sub-linear privacy loss with a sub-linear regret while using the original EXP3. Here, we will prove that an existing technique is already achieving this goal. The algorithm which we called ${\emph{EXP3}}_{\tau}$ is from (?). It groups the rounds into disjoint intervals of fixed size $\tau$ where the $j$’th interval starts on round $(j-1)\tau+1$ and ends on round $j\tau$. At the beginning of interval $j$, ${\emph{EXP3}}_{\tau}$ receives an action from EXP3 and plays it for $\tau$ rounds. During that time, EXP3 does not observe any feedback. At the end of the interval, ${\emph{EXP3}}_{\tau}$ feeds EXP3 with a single gain, the average gain received during the interval.

Theorem 3.4 borrowed from (?) specifies the upper bound on the regret ${\emph{EXP3}}_{\tau}$. It is remarkable that this bound holds against the m-memory bounded adaptive adversary. While in theorem 3.5, we show the privacy loss enjoyed by this algorithm, one gets a better intuition of how good those results are from corollary 3.2 and 3.3. Indeed, we can observe that ${\emph{EXP3}}_{\tau}$ achieves a sub-logarithmic privacy loss of $\mathcal{O}(\sqrt{\ln T})$ with a regret of $\mathcal{O}(T^{2/3})$ against a special case of the m-memory bounded adaptive adversary called the switching costs adversary for which $m=1$. This is the optimal regret bound (in the sense that there is a matching lower bound (?)). This means that in some sense we are getting privacy for free against this adversary.

Theorem 3.4** (Regret of ${\emph{EXP3}}_{\tau}$ (?)).**

The expected regret of ${\emph{EXP3}}_{\tau}$ is upper bounded by:

[TABLE]

against the m-memory bounded adaptive adversary for any $m<\tau$.

Theorem 3.5** (Privacy loss of ${\emph{EXP3}}_{\tau}$).**

${\emph{EXP3}}_{\tau}$* is $\left(\frac{4T}{\tau^{3}}+\sqrt{8\ln(1/\delta^{\prime})\frac{T}{\tau^{3}}},\delta^{\prime}\right)$-DP up to round $T$.*

Proof.

The sensitivity of each gain is now $\frac{1}{\tau}$ as we are using the average. Combined with theorem (3.3), it means the per-round privacy loss is $2\frac{T}{\tau}$. Given that EXP3 only observes $\frac{T}{\tau}$ rounds, using the advanced composition theorem (?) (Theorem III.3) concludes the final privacy loss over $T$ rounds. ∎

Corollary 3.2**.**

${\emph{EXP3}}_{\tau}$* run with $\tau=(7K\log K)^{-1/3}T^{1/3}$ is $(\epsilon,\delta^{\prime})$ differentially private up to round $T$ with $\delta^{\prime}=T^{-2}$, $\epsilon=28K\ln K+\sqrt{112K\ln K\ln T}$. Its expected regret against the switching costs adversary is upper bounded by $2(7K\ln K)^{1/3}T^{2/3}+(7K\log K)^{-1/3}T^{1/3}$.*

Proof.

The proof is immediate by replacing $\tau$ and $\delta^{\prime}$ in Theorem 3.4 and 3.5 and the fact that for the switching costs adversary, $m=1$. ∎

Corollary 3.3**.**

${\emph{EXP3}}_{\tau}$* run with $\tau=\left(\frac{4T\epsilon+2T\ln\frac{1}{\delta}}{\epsilon^{2}}\right)^{1/3}$ is $(\epsilon,\delta)$ differentially private and its expected regret against the switching costs adversary is upper bounded by: $\mathcal{O}\left({T^{2/3}\sqrt{K\ln K}\left(\frac{\sqrt{\ln\frac{1}{\delta}}}{\epsilon}\right)^{1/3}}\right)$*

4 Experiments

We tested DP-EXP3-Lap, ${\textsc{EXP3}}{}_{\tau}$ together with the non-private EXP3 against a few different adversaries. The privacy parameter $\epsilon$ of DP-EXP3-Lap is set as defined in corollary 3.2. This is done so that the regret of DP-EXP3-Lap and ${\textsc{EXP3}}{}_{\tau}$ are compared with the same privacy level. All the other parameters of DP-EXP3-Lap are taken as defined in corollary 3.1 while the parameters of ${\textsc{EXP3}}{}_{\tau}$ are taken as defined in corollary 3.2.

For all experiments, the horizon is $T=2^{18}$ and the number of arms is $K=4$. We performed $720$ independent trials and reported the median-of-means estimator222Used heavily in the streaming literature (?) of the cumulative regret. It partitions the trials into $a_{0}$ equal groups and return the median of the sample means of each group. Proposition 4.1 is a well known result (also in (?; ?)) giving the accuracy of this estimator. Its convergence is $\mathcal{O}(\sigma/\sqrt{N})$, with exponential probability tails, even though the random variable $x$ may have heavy-tails. In comparison, the empirical mean can not provide such guarantee for any $\sigma>0$ and confidence in $[0,1/(2e)]$ (?).

Proposition 4.1*.*

Let $x$ be a random variable with mean $\mu$ and variance $\sigma^{2}<\infty$. Assume that we have $N$ independent sample of $x$ and let $\hat{\mu}$ be the median-of-means computed using $a_{0}$ groups. With probability at least $1-e^{-a_{0}/4.5}$, $\hat{\mu}$ satisfies $\left|\hat{\mu}-\mu\right|\leq\sigma\sqrt{6a_{0}/N}$.

We set the number of groups to $a_{0}={24}{}$, so that the confidence interval holds w.p. at least $0.995$.

We also reported the deviation of each algorithm using the Gini’s Mean Difference (GMD hereafter) (?). GMD computes the deviation as $\sum_{j=1}^{N}(2j-N-1)x_{(j)}$ with $x_{(j)}$ the $j$-th order statistics of the sample (that is $x_{(1)}\leq x_{(2)}\leq\ldots\leq x_{(N)}$). As shown in (?; ?), the GMD provides a superior approximation of the true deviation than the standard one. To account for the fact that the cumulative regret of our algorithms might not follow a symmetric distribution, we computed the GMD separately for the values above and below the median-of-means.

At round $t$, we computed the cumulative regret against the fixed oracle who plays the best arm assuming that the end of the game is at $t$. The oracle uses the actual sequence of gains to decide his best arm. For a given trial, we make sure that all algorithms are playing the same game by generating the gains for all possible pair of round-arm before the game starts.

Deterministic adversary.

As shown by (?), the expected regret of any agent against an oblivious adversary can not be worse than that against the worst case deterministic adversary. In this experiment, arm $2$ is the best and gives $1$ for every even round. To trick the players into picking the wrong arms, the first arm always gives $0.38$ whereas the third gives $1$ for every round multiple of $3$. The remaining arms always give [math]. As shown by the figure, this simple adversary is already powerful enough to make the algorithms attain their upper bound.

Stochastic adversary

This adversary draws the gains of the first arm i.i.d from $\mathop{\mathpzc{Bern}}\nolimits\left({0.55}\right)$ whereas all other gains are drawn i.i.d from $\mathop{\mathpzc{Bern}}\nolimits\left({0.5}\right)$.

Fully oblivious adversary.

For the best arm $k$, it first draws a number $p$ uniformly in $[0.5,0.5+2\cdot\varepsilon]$ and generates the gain $g_{k,t}\sim\mathop{\mathpzc{Bern}}\nolimits\left({p}\right)$. For all other arms, $p$ is drawn from $[0.5-\varepsilon,0.5+\varepsilon]$. This process is repeated at every round. In our experiments, $\varepsilon=0.05$

An oblivious adversary.

This adversary is identical to the fully oblivious one for every round multiple of $200$. Between two multiples of $200$ the last gain of the arm is given.

The Switching costs adversary

This adversary (defined at Figure 1 in (?)) defines a stochastic processes (including simple Gaussian random walk as special case) for generating the gains. It was used to prove that any algorithm against this adversary must incur a regret of $\mathcal{O}(T^{2/3})$.

Discussion

Figure 1 shows our results against a variety of adversaries, with respect to a fixed oracle. Overall, the performance (in term of regret) of DP-EXP3-Lap is very competitive against that of EXP3 while providing a significant better privacy. This means that DP-EXP3-Lap allows us to get privacy for free in the bandit setting against an adversary not more powerful than the oblivious one.

The performance of ${\textsc{EXP3}}{}_{\tau}$ is worse than that of DP-EXP3-Lap against an oblivious adversary or one less powerful. However, the situation is completely reversed against the more powerful switching cost adversary. In that setting, ${\textsc{EXP3}}{}_{\tau}$ outperforms both EXP3 and DP-EXP3-Lap confirming the theoretical analysis. We can see ${\textsc{EXP3}}{}_{\tau}$ as the algorithm providing us privacy for free against switching cost adversary and adaptive m-bounded memory one in general.

5 Conclusion

We have provided the first results on differentially private adversarial multi-armed bandits, which are optimal up to logarithmic factors. One open question is how differential privacy affects regret in the full reinforcement learning problem. At this point in time, the only known results in the MDP setting obtain differentially private algorithms for Monte Carlo policy evaluation (?). While this implies that it is possible to obtain policy iteration algorithms, it is unclear how to extend this to the full online reinforcement learning problem.

Acknowledgements.

This research was supported by the SNSF grants “Adaptive control with approximate Bayesian computation and differential privacy” and “Swiss Sense Synergy”, by the Marie Curie Actions (REA 608743), the Future of Life Institute “Mechanism Design for AI Architectures” and the CNRS Specific Action on Security.