Near-Optimal Algorithms for Private Online Optimization in the Realizable Regime

TL;DR

This paper introduces new differentially private algorithms for online learning in the realizable setting, achieving near-optimal regret bounds that significantly improve over previous non-realizable results.

Contribution

It presents the first near-optimal DP algorithms for online prediction and convex optimization in the realizable regime, with improved regret bounds.

Findings

Expert prediction regret: O(ε^{-1} log^{1.5} d)

Adaptive small-loss regret: O(L* log d + ε^{-1} log^{1.5} d)

Convex optimization regret: O(ε^{-1} d^{1.5}) and O(ε^{-2/3} (dT)^{1/3})

Abstract

We consider online learning problems in the realizable setting, where there is a zero-loss solution, and propose new Differentially Private (DP) algorithms that obtain near-optimal regret bounds. For the problem of online prediction from experts, we design new algorithms that obtain near-optimal regret ${O} \big( \varepsilon^{-1} \log^{1.5}{d} \big)$ where $d$ is the number of experts. This significantly improves over the best existing regret bounds for the DP non-realizable setting which are ${O} \big( \varepsilon^{-1} \min\big\{d, T^{1/3}\log d\big\} \big)$. We also develop an adaptive algorithm for the small-loss setting with regret $O(L^\star\log d + \varepsilon^{-1} \log^{1.5}{d})$ where $L^\star$ is the total loss of the best expert. Additionally, we consider DP online convex optimization in the realizable setting and propose an algorithm with near-optimal regret $O \big(\varepsilon^{-1} d^{1.5} \big)$, as well as an algorithm for the smooth case with regret $O \big( \varepsilon^{-2/3} (dT)^{1/3} \big)$, both significantly improving over existing bounds in the non-realizable regime.

Tables

Table 1. Table 1: Comparison between (normalized) regret upper bounds for the realizable and non-realizable case for both DP-OPE and DP-OCO. For readability, we omit logarithmic factors in T 𝑇 T and 1 / δ 1 𝛿 1/\delta .

	Non-realizable	Realizable (This work)
DP-OPE	$\min \{\frac{\sqrt{d}}{T\epsilon },\sqrt{\frac{\log d}{T}}+\frac{\log d}{T^{2/3}\epsilon }\}$ [AS17, AFKT22]	$\frac{{\log }^{1.5}d}{T\epsilon }$
DP-OCO	${\left(\frac{\sqrt{d}}{T\epsilon }\right)}^{1/2}$ [KMSTTX21]	$\frac{d^{1.5}}{T\epsilon }$
DP-OCO (smooth)	${\left(\frac{\sqrt{d}}{T\epsilon }\right)}^{1/2}$ [KMSTTX21]	${\left(\frac{\sqrt{d}}{T\epsilon }\right)}^{2/3}$

Equations

$$Re{g}_T(\mathcal{A})=\sum _{t=1}^T{\ell }_t(x_t)-\underset{x^{\star }\in \mathcal{X}}{\min }\sum _{t=1}^T{\ell }_t(x^{\star }).$$

$$\mathrm{Pr}[\mathcal{A}(\mathcal{S})\in \mathcal{O}]\le e^{\epsilon }\mathrm{Pr}[\mathcal{A}({\mathcal{S}}^{\prime })\in \mathcal{O}]+\delta .$$

$$\Big{|}c_{t}-\sum_{i=1}^{t}a_{i}\Big{|}=\frac{1}{\varepsilon}\cdot\mathsf{poly}(\log(\beta^{-1})\log{T}).$$

$$\sum _{t=1}^T{\ell }_t(x_t)\le O\left(L^{\star }\log (d/\beta )+\frac{{\log }^2(d)+\log (T/\beta )\log (d/\beta )}{\epsilon }\right).$$

$$\sum _{t=1}^T{\ell }_t(x_t)\le O\left(L^{\star }\log (d/\beta )+\frac{{\log }^{3/2}(d)\sqrt{\log (1/\delta )}+\log (T/\beta )\log (d/\beta )}{\epsilon }\right).$$

$$\mathbb{P}(x_t=x)\propto e^{-\eta s_t(x)/2}$$

$${\varphi }_t=\sum _{x\in [d]}{e}^{-\eta L_t(x)/2},$$

$$P(Z_i=k+1)\le (2/3{)}^k.$$

$$P(x_t\text{ is switched before ti+1})$$

$$=\sum _{x\in [d]}\frac{e^{-\eta L_t(x)/2}}{{\varphi }_t}\cdot 1\left\{L_{t_{i+1}}(x)-L_t(x)\ge 4/\eta \right\}$$

$$\le \sum _{x\in [d]}\frac{e^{-\eta L_t(x)/2}}{{\varphi }_t}\cdot \frac{1-e^{-\eta (L_{t_{i+1}}(x)-L_t(x))/2}}{1-e^{-2}}$$

$$\le 4/3(1-{\varphi }_{t_{i+1}}/{\varphi }_t)$$

$$\le 2/3.$$

$$P(Z\ge 6m+24\log (1/\beta ))\le \beta .$$

$$\sum _{t=1}^T{\ell }_t(x_t)\le O\left(L^{\star }\log (d/\beta )\log (T)+\frac{{\log }^2(d)\log (T)+\log (T/\beta )\log (d/\beta )\log (T)}{\epsilon }\right).$$

$$O\left({\bar{L}}^{\star }\log (d/\beta )+\frac{{\log }^2(d)+\log (T/\beta )\log (d/\beta )}{{\epsilon }_0}\right).$$

$$O\left({\bar{L}}^{\star }\log (d/\beta )+\frac{{\log }^2(d)+\log (T/\beta )\log (d/\beta )}{{\epsilon }_0}\right),$$

$$O\left(L^{{}^{\star }}d\log (LD/\beta )+\frac{d^{3/2}{\log }^{3/2}(LDT)\sqrt{\log (1/\delta )}+\log (T/\beta )d\log (LD/\beta )}{\epsilon }\right).$$

$$\sum _{t=1}^T{\ell }_t(x_t)-L_{\mathsf{experts}}^{{}^{\star }}\le O\left(L_{\mathsf{experts}}^{{}^{\star }}\log (M/\beta )+\frac{{\log }^2(M)+\log (T/\beta )\log (M/\beta )}{\epsilon }\right)$$

$$|L^{\star }-L_{\mathsf{experts}}^{{}^{\star }}|=|\underset{x\in \mathcal{X}}{\min }\sum _{t=1}^T{\ell }_t(x)-\underset{x\in {\mathcal{X}}_{\mathsf{experts}}^{\rho }}{\min }\sum _{t=1}^T{\ell }_t(x)|\le TL\rho .$$

$$\sum _{t=1}^T{\ell }_t(x_t)-L^{{}^{\star }}\le O\left((L^{{}^{\star }}+TL\rho )\log (M/\beta )+\frac{{\log }^{3/2}(M)\sqrt{\log (1/\delta )}+\log (T/\beta )\log (M/\beta )}{\epsilon }\right).$$

$$x_t={\mathrm{argmin}}_{x\in \mathcal{X}}⟨{\bar{g}}_{t-1},x⟩+\frac{\lambda }{2}{\Vert x\Vert }_2^2,$$

$$\frac{1}{T}\mathbb{E}\left[\sum _{t=1}^T{\ell }_t(x_t)-{\ell }_t(x^{\star })\right]$$

$$x_{t+1}={\mathrm{argmin}}_{x\in \mathcal{X}}\sum _{i=1}^t⟨\nabla {\ell }_i(x_i),x⟩+\frac{\lambda }{2}{\Vert x\Vert }_2^2+⟨b_t,x⟩,$$

$${\hat{x}}_{t+1}={\mathrm{argmin}}_{x\in \mathcal{X}}\sum _{i=1}^t⟨\nabla {\ell }_i(x_i),x⟩+\frac{\lambda }{2}{\Vert x\Vert }_2^2.$$

$$\sum _{t=1}^T{\ell }_t(x_t)-{\ell }_t(x^{\star })$$

$$=\sum _{t=1}^T⟨\nabla {\ell }_t(x_t),x_t-{\hat{x}}_t⟩+\sum _{t=1}^T⟨\nabla {\ell }_t(x_t),{\hat{x}}_t-x^{\star }⟩$$

$$\le \sum _{t=1}^T{\Vert \nabla {\ell }_t(x_t)\Vert }_2{\Vert x_t-{\hat{x}}_t\Vert }_2+\sum _{t=1}^T⟨\nabla {\ell }_t(x_t),{\hat{x}}_t-x^{\star }⟩$$

$$\le \frac{1}{8\beta }\sum _{t=1}^T{\Vert \nabla {\ell }_t(x_t)\Vert }_2^2+4\beta \sum _{t=1}^T{\Vert x_t-{\hat{x}}_t\Vert }_2^2+\sum _{t=1}^T⟨\nabla {\ell }_t(x_t),{\hat{x}}_t-x^{\star }⟩$$

$$\le \frac{1}{2}\sum _{t=1}^T{\ell }_t(x_t)+4\beta \sum _{t=1}^T{\Vert b_t\Vert }_2^2/{\lambda }^2+\sum _{t=1}^T⟨\nabla {\ell }_t(x_t),{\hat{x}}_t-x^{\star }⟩,$$

$$\sum _{t=1}^T⟨\nabla {\ell }_t(x_t),{\hat{x}}_t-x^{\star }⟩$$

$$\le \frac{8\beta }{\lambda }\sum _{t=1}^T{\ell }_t(x_t)+\lambda D^2.$$

$$\sum _{t=1}^T{\ell }_t(x_t)-{\ell }_t(x^{\star })$$

$$\mathbb{E}[{\Vert b_t\Vert }_2^2]\le O\left(\frac{L^{2}d\log (T)\log (1/\delta )}{{\epsilon }^2}\right)$$

Full text

1 Introduction

We study the problem of private online optimization in the realizable setting where there is a zero-loss solution. In this problem, an online algorithm $\mathcal{A}$ interacts with an adversary over $T$ rounds. The adversary picks a (non-negative) loss function $\ell_{t}:\mathcal{X}\to\mathbb{R}$ at round $t$ and simultaneously the algorithm $\mathcal{A}$ picks a response $x_{t}$, suffering loss $\ell_{t}(x_{t})$. The algorithm aims to minimize the regret, which is the loss compared to the best solution $x^{\star}\in\mathcal{X}$ in hindsight, while at the same time keeping the sequence of predictions $x_{1},\ldots,x_{T}$ differentially private with respect to individual loss functions.

In this paper, we focus on two well-studied instances of this problem. In differentially private online prediction from experts (DP-OPE), we have $d$ experts $\mathcal{X}=[d]$ and the adversary chooses a loss function $\ell_{t}:[d]\to[0,1]$. Our second setting is differentially private online convex optimization (DP-OCO) where $\mathcal{X}\subset\mathbb{R}^{d}$ is a convex set with bounded diameter, and the adversary chooses convex and $L$-Lipschitz loss functions $\ell_{t}:\mathcal{X}\to\mathbb{R}^{+}$.

Several papers have recently studied DP-OPE and DP-OCO in the general non-realizable setting [JKT12, ST13, JT14, AS17, KMSTTX21]. These papers have resulted in different algorithms with sub-linear regret for both problems. For DP-OPE,  [AS17, JT14] developed private versions of follow-the-regularized-leader (FTRL) obtaining (normalized) regret \min\big{\{}{d}/{T\varepsilon},{\sqrt{T\log d}}/{T\varepsilon}\big{\}}. More recently, [AFKT22] developed low-switching algorithms for DP-OPE with oblivious adversaries, obtaining normalized regret roughly $O(\sqrt{\log(d)/T}+\log d/T^{2/3}\varepsilon)$. Additionally, for the problem of DP-OCO, [KMSTTX21] have recently proposed a DP-FTRL algorithm based on the binary tree mechanism which obtains (normalized) regret \big{(}{\sqrt{d}}/{T\varepsilon}\big{)}^{1/2}.

Despite this progress, the regret bounds of existing algorithms are still polynomially worse than existing lower bounds. Currently, the only existing lower bounds for oblivious adversaries are the trivial bounds from the non-online versions of the same problems: for DP-OPE, lower bounds for private selection [SU17] imply a (normalized) regret lower bound of $O({\log(d)}/{T\varepsilon)}$, while existing lower bounds for DP-SCO [FKT20] give a (normalized) regret lower bound of $\Omega({\sqrt{d}}/{T\varepsilon})$ for DP-OCO.

Practical optimization problems arising from over-parameterized models often lead to instances that additionally satisfy realizability, i.e. that the optimal loss is zero or close to zero. This motivates the study of designing algorithms that can do better under this assumption. Realizability has been studied since the early days of learning theory and ubiquitous in the non-private online optimization literature [SST10, SS12, Haz16]. It has proven useful for improving regret bounds in non-private OPE and OCO [SS12, SST10] and in the closely related problem of differentially private stochastic convex optimization (DP-SCO) [ACCD22]. In this work we study DP-OPE and DP-OCO in the realizable setting and develop new algorithms that obtain near-optimal regret bounds in several settings.

1.1 Contributions

We propose new algorithms and lower bounds for the problems of differentially private online prediction from experts (DP-OPE) and differentially private online convex optimization (DP-OCO) in the realizable setting. The following are our primary contributions:

• •

Near-optimal algorithms for DP-OPE.   We design new algorithms that obtain near-optimal regret $\widetilde{O}\left(\log^{1.5}(d)/\varepsilon\right)$ for DP-OPE with $d$ experts when there is a zero-loss expert. The best existing algorithms for non-realizable DP-OPE obtain significantly worse regret bounds \min\big{\{}{d}/{\varepsilon},T^{1/3}\log d/{\varepsilon}\big{\}} [AS17, AFKT22], which have a polynomial dependence on either $T$ or the number of experts $d$. Our algorithms build on sequential applications of the exponential mechanism to pick a good expert, and the sparse-vector-technique to identify when the current expert is no longer a good expert (with near-zero loss). Crucially, an oblivious adversary cannot identify which expert the algorithm has picked, resulting in a small number of switches. We deploy a potential-based proof strategy to show that this algorithm have logarithmic number of switches. We also show that a lower bound of $\Omega(\log d/\varepsilon)$ holds for any $\varepsilon$-DP algorithm even in the realizable case.

• •

Adaptive algorithms for DP-OPE with low-loss experts.   We also develop an algorithm that adapts to the setting where there is an expert with low loss, that is, $L^{\star}=\min_{x\in[d]}\sum_{t=1}^{T}\ell_{t}(x)$. Our algorithms are adaptive to the value of $L^{\star}$ and obtain total regret of $L^{\star}\log d+\varepsilon^{-1}\log^{1.5}d$.

• •

Near-optimal regret for low-dimensional DP-OCO.   Building on our algorithms for DP-OPE, we propose a new algorithm for DP-OCO that obtains regret $\widetilde{O}\left(d^{1.5}/\varepsilon\right)$. This is near-optimal for low-dimensional problems where $d=O(1)$ and improves over the best existing algorithm which obtains a normalized regret $(\sqrt{d}/T\varepsilon)^{1/2}$ [KMSTTX21].

• •

Improved regret for smooth DP-OCO.   When the loss function is smooth, we show that DP-FTRL [KMSTTX21] with certain parameters obtains an improved normalized regret of $(\sqrt{d}/T\varepsilon)^{2/3}$ if there is a zero-loss expert.

1.2 Related work

Several works have studied online optimization in the realizable setting, developing algorithms with better regret bounds [SS12, SST10]. For online prediction from experts, the weighted majority algorithm obtains a regret bound of $4\log{d}$ compared to $O(\sqrt{T\log d})$ in the non-realizable setting. Moreover, for online convex optimization, [SST10] show that online mirror descent achieves regret $4\beta D^{2}+2\sqrt{\beta D^{2}TL^{\star}}$ compared to $O(\sqrt{T})$ in the general case.

On the other hand, the private online optimization literature has mainly studied the general non-realizable case [JKT12, ST13, JT14, AS17, KMSTTX21]. For online prediction from experts, the best existing regret bounds for $(\varepsilon,\delta)$-DP are $O(\varepsilon^{-1}\sqrt{T\log d\log(1/\delta)})$ [JT14] and $O(\sqrt{T\log d}+\varepsilon^{-1}\sqrt{d\log(1/\delta)}\log d\log^{2}T)$ [AS17]. [AFKT22] show that these rates can be improved using a private version of the shrinking dartboard algorithm, obtaining regret roughly $O(\sqrt{T\log d}+T^{1/3}\log d/\varepsilon)$. For online convex optimization, [KMSTTX21] developed a private follow-the-regularized-leader algorithm using the binary tree mechanism that obtains normalized regret bound \widetilde{O}\big{(}{\sqrt{d}}/{T\varepsilon}\big{)}^{1/2}.

The realizable setting has recently been studied in the different but related problem of differentially private stochastic convex optimization (DP-SCO) [ACCD22]. DP-SCO and DP-OCO are closely related as one can convert an OCO algorithm into an SCO algorithm using standard online-to-batch transformations [Haz16] [ACCD22] study DP-SCO problems in the interpolation regime where there exists a minimizer that minimizes all loss functions, and propose algorithms that improve the regret over the general setting if the functions satisfy certain growth conditions.

2 Preliminaries

In online optimization, we have an interactive $T$-round game between an adversary and an online algorithm. In this paper, we focus on oblivious adversaries that choose in advance a sequence of loss functions $\ell_{1},\dots,\ell_{T}$ where $\ell_{t}:\mathcal{X}\to\mathbb{R}$. Then, at round $t$, the adversary releases a loss function $\ell_{t}$ and simultaneously the algorithm plays a solution $x_{t}\in\mathcal{X}$. The algorithm then suffers loss $\ell_{t}(x_{t})$ at this round. The regret of the online algorithm is

[TABLE]

For ease of notation, for an oblivious adversary that chooses a loss sequence $\mathcal{S}=(\ell_{1},\dots,\ell_{T})$, we let $\mathcal{A}(\mathcal{S})=(x_{1},\dots,x_{T})$ denote the output of the interaction between the online algorithm and the adversary.

In this work, we are mainly interested in two instances of the above general online optimization problem:

• •

Online prediction from experts (OPE).   In this problem, we have a set of $d$ experts $\mathcal{X}=[d]$, and the adversary chooses loss functions $\ell_{t}:[d]\to[0,1]$.

• •

Online convex optimization (OCO).   In OCO, we are optimizing over a convex set $\mathcal{X}\subseteq\mathbb{R}^{d}$ with bounded diameter $\mathsf{diam}(\mathcal{X})\leq D$,111The diameter of a set $\mathcal{X}\subseteq\mathbb{R}^{d}$ (in Euclidean geometry) is defined as $\mathsf{diam}(\mathcal{X})=\sup_{x,y\in\mathcal{X}}\|x-y\|$. and the adversary chooses loss functions $\ell_{t}:\mathcal{X}\to\mathbb{R}$ that are convex and $L$-Lipschitz.

We are mainly interested in the so-called realizable setting. More precisely, we say than an OPE (or OCO) problem is realizable if there exists a feasible solution $x^{\star}\in\mathcal{X}$ such that $L^{\star}=\sum_{t=1}^{T}\ell_{t}(x^{\star})=0$. We also extend some of our results to the near-realizable setting where $0<L^{\star}\ll T$.

The main goal of this paper is to study both of these problems under the restriction of differential privacy.

Definition 2.1** (Differential Privacy).**

A randomized algorithm $\mathcal{A}$ is $(\varepsilon,\delta)$-differentially private against oblivious adversaries ($(\varepsilon,\delta)$-DP) if, for all sequences $\mathcal{S}=(\ell_{1},\dots,\ell_{T})$ and $\mathcal{S}^{\prime}=(\ell^{\prime}_{1},\dots,\ell^{\prime}_{T})$ that differ in a single element, and for all events $\mathcal{O}$ in the output space of $\mathcal{A}$, we have

[TABLE]

We note that our algorithms satisfy a stronger privacy guarantee against adaptive adversaries (see for example the privacy definition in [JRSS21]). However, we choose to focus solely on oblivious adversaries for ease of presentation and readability.

2.1 Background on Differential Privacy

In our analysis, we require the following standard privacy composition result.

Lemma 2.1** (Advanced composition [DR14]).**

If $\mathcal{A}_{1},\dots,A_{k}$ are randomized algorithms that each is $(\varepsilon,\delta)$-DP, then their composition $(\mathcal{A}_{1}(\mathcal{S}),\dots,A_{k}(\mathcal{S}))$ is $(\sqrt{2k\log(1/\delta^{\prime})}\varepsilon+k\varepsilon(e^{\varepsilon}-1),\delta^{\prime}+k\delta)$-DP.

In addition to basic facts about differential privacy such as composition and post-processing, our development uses two key techniques from the privacy literature: the Sparse-vector-technique and the binary tree mechanism, which we now describe.

Sparse vector technique.

We recall the sparse-vector-technique [DR14] which we use for the realizable setting in Section 3. Given an input $\mathcal{S}=(z_{1},\dots,z_{n})\in\mathcal{Z}^{n}$, the algorithm takes a stream of queries $q_{1},q_{2},\dots,q_{T}$ in an online manner. We assume that each $q_{i}$ is $1$-sensitive, that is, $|q_{i}(\mathcal{S})-q_{i}(\mathcal{S}^{\prime})|\leq 1$ for neighboring datasets $\mathcal{S},\mathcal{S}^{\prime}\in\mathcal{Z}^{n}$ that differ in a single element. We have the following guarantee.

Lemma 2.2** ([DR14], Theorem 3.24).**

Let $\mathcal{S}=(z_{1},\dots,z_{n})\in\mathcal{Z}^{n}$. For a threshold $L$ and $\beta>0$, there is an $\varepsilon$-DP algorithm ($\mathsf{AboveThreshold}$) that halts at time $k\in[T+1]$ such that for $\alpha=\frac{8(\log T+\log(2/\beta))}{\varepsilon}$ with probability at least $1-\beta$,

• •

For all $t<k$, $q_{i}(\mathcal{S})\leq L+\alpha$;

• •

$q_{k}(\mathcal{S})\geq L-\alpha$* or $k=T+1$.*

To facilitate the notation for using $\mathsf{AboveThreshold}$ in our algorithms, we assume that it has the following components:

$\mathsf{InitializeSparseVec}(\varepsilon,L,\beta)$: initializes a new instance of $\mathsf{AboveThreshold}$ with privacy parameter $\varepsilon$, threshold $L$, and probability parameter $\beta$. This returns an instance (data structure) $Q$ that supports the following two functions. 2. 2.

$Q.\mathsf{AddQuery}(q)$: adds a new query $q:\mathcal{Z}^{n}\to\mathbb{R}$ to $Q$. 3. 3.

$Q.\mathsf{TestAboThr}()$: tests if the last query that was added to $Q$ was above threshold. In that case, the algorithm stops and does not accept more queries.

The binary tree mechanism.

We also build on the binary tree mechanism [DNPR10, CSS11] which allows to privately estimate the running sum of a sequence of $T$ numbers $a_{1},\dots,a_{T}\in[0,1]$.

Lemma 2.3** ([DNPR10], Theorem 4.1).**

Let $\varepsilon\leq 1$. There is an $\varepsilon$-DP algorithm (BinaryTree) that takes a stream of numbers $a_{1},a_{2},\dots,a_{T}$ and outputs $c_{1},c_{2},\dots,c_{T}$ such that for all $t\in[T]$ with probability at least $1-\beta$,

[TABLE]

The same approach extends to the case when $a_{i}$’s are vectors in $\mathbb{R}^{d}$ with $\|a_{i}\|_{2}\leq 1$. In this case, the error vector $(c_{t}-\sum_{i=1}^{t}a_{i})$ is distributed at $\mathcal{N}(0,d\cdot\mathsf{poly}(\log T/\beta\delta)/\varepsilon^{2}\mathbb{I})$ and the mechanism satisfies $(\varepsilon,\delta)$-DP.

Additional notation.

For a positive integer $k\in\mathbb{N}$, we let $[k]=\{1,2,\dots,k\}$. Moreover, for a sequence $a_{1},\dots,a_{t}$, we use the shorthand $a_{1:t}=a_{1},\dots,a_{t}$.

3 Near-optimal regret for online prediction from experts

In this section, we consider the online prediction from experts problem in the near-realizable regime, where the best expert achieves small loss $L^{\star}llT$. Under this setting, we develop a new private algorithm that achieves regret $\widetilde{O}(L^{\star}\log d+\log^{3/2}(d)/\varepsilon)$. For the realizable setting where $L^{\star}=0$, this algorithm obtains near-optimal regret $\widetilde{O}(\log^{3/2}(d)/\varepsilon)$.

The algorithm builds on the fact that an oblivious adversary cannot know which expert the algorithm picks. Therefore, if the algorithm picks a random good expert with loss smaller than $L^{\star}$, the adversary has to increase the loss for many experts before identifying the expert chosen by the algorithm. The algorithm will therefore proceed as follows: at each round, privately check using sparse-vector-technique whether the previous expert is still a good expert (has loss nearly $L^{\star}$). If not, randomly pick (privately) a new expert from the set of remaining good experts. The full details are in Algorithm 1.

The following theorem summarizes the performance of Algorithm 1.

Theorem 1**.**

Let $\ell_{1},\dots,\ell_{T}\in[0,1]^{d}$ be chosen by an oblivious adversary such that there is $x^{\star}\in[d]$ such that $\sum_{t=1}^{T}\ell_{t}(x^{\star})\leq L^{\star}$. Let $0<\beta<1/2$, $B=\log(2T^{2}/\beta)$, $K=6\left\lceil{\log d}\right\rceil+24\log(1/\beta)$, and $L=L^{\star}+4/\eta+\frac{8B}{\varepsilon}$. If $\eta=\varepsilon/2K$ then Algorithm 1 is $\varepsilon$-DP and with probability at least $1-O(\beta)$ has regret

[TABLE]

Further, if $\varepsilon\leq\sqrt{\log T\log(1/\delta)}$ and $\eta=\varepsilon/4\sqrt{2K\log(1/\delta)}$ then Algorithm 1 is $(\varepsilon,\delta)$-DP and with probability at least $1-O(\beta)$ has regret

[TABLE]

While Algorithm 1 requires the knowledge of $L^{\star}$, we also design an adaptive version that does not require $L^{\star}$ in the next section. Note that the algorithm obtains regret roughly $\log^{3/2}(d)/\varepsilon$ for the realizable setting where $L^{\star}=0$.

Proof.

First, we prove the privacy guarantees of the algorithm using privacy composition results: there are $K$ applications of the exponential mechanism with privacy parameter $\eta$. Moreover, sparse-vector is applied over each user’s data only once, hence the $K$ applications of sparse-vector are $\varepsilon/2$-DP. Overall, the algorithm is $(\varepsilon/2+K\eta)$-DP and $(\varepsilon/2+\sqrt{2K\log(1/\delta)}\eta+K\eta(e^{\eta}-1),\delta)$-DP (using advanced compositions; see Lemma 2.1). Setting $\eta=\varepsilon/2K$ results in $\varepsilon$-DP and $\eta=O(\varepsilon/\sqrt{K\log(1/\delta)})$ results in $(\varepsilon,\delta)$-DP.

We proceed to analyze utility. First, note that the guarantees of the sparse-vector algorithm (Lemma 2.2) imply that with probability at least $1-\beta$ for each time-step $t\in[T]$, if sparse-vector identifies above threshold query then s_{t}(x)\geq\underaccent{\bar}{\Delta}\coloneqq L-\frac{8B}{\varepsilon}\geq 4/\eta. Otherwise, $s_{t}(x)\leq\bar{\Delta}\coloneqq L+\frac{8B}{\varepsilon}$. In the remainder of the proof, we condition on this event. The idea is to show that the algorithm has logarithmic number of switches, and each switch the algorithm pays roughly $1/\varepsilon$ regret.

To this end, we define a potential at time $t\in[T]$:

[TABLE]

where $L_{t}(x)=\max(\sum_{j=1}^{t-1}\ell_{j}(x),L^{\star})$. Note that $\phi_{1}=de^{-\eta L^{\star}/2}$ and $\phi_{t}\geq e^{-\eta L^{\star}/2}$ for all $t\in[T]$ as there is $x\in[d]$ such that $\sum_{t=1}^{T}\ell_{t}(x)=L^{\star}$. We split the iterates to $m=\left\lceil{\log d}\right\rceil$ rounds $t_{0},t_{1},\dots,t_{m}$ where $t_{i}$ is the largest $t\in[T]$ such that $\phi_{t_{i}}\geq\phi_{1}/2^{i}$. Let $Z_{i}$ be the number of switches in $[t_{i},t_{i+1}-1]$ (number of times the exponential mechanism is used to pick $x_{t}$). The following key lemma shows that $Z_{i}$ cannot be too large.

Lemma 3.1**.**

Fix $0\leq i\leq m-1$. Then for any $1\leq k\leq T$, it holds that

[TABLE]

Proof.

Let $t_{i}\leq t\leq t_{i+1}$ be a time-step where a switch happens (exponential mechanism is used to pick $x_{t}$). Note that $\phi_{t_{i+1}}\geq\phi_{t}/2$. We prove that the probability that $x_{t}$ is switched between $t$ and $t_{i+1}$ is at most $2/3$. To this end, note that if $x_{t}$ is switched before $t_{i+1}$ then \sum_{i=t}^{t_{i+1}}\ell_{i}(x)\geq\underaccent{\bar}{\Delta} as sparse-vector identifies $x_{t}$, and therefore L_{t_{i+1}}(x)-L_{t}(x)\geq\underaccent{\bar}{\Delta}-L^{\star}\geq 4/\eta. Thus we have that

[TABLE]

where the second inequality follows the fact that $1\!\left\{a\geq b\right\}\leq\frac{1-e^{-\eta b}}{1-e^{-\eta a}}$ for $a,b,\eta\geq 0$, and the last inequality since $\phi_{t_{i+1}}/\phi_{t_{1}}\geq 1/2$. This argument shows that after the first switch inside the range $[t_{i},t_{i+1}]$, each additional switch happens with probability at most $2/3$. The claim follows. ∎

We now proceed with the proof. Let $Z=\sum_{i=0}^{m-1}Z_{i}$ be the total number of switches. Note that $Z\leq m+\sum_{i=0}^{m-1}\max(Z_{i}-1,0)$ and Lemma 3.1 implies $\max(Z_{i}-1,0)$ is upper bounded by a geometric random variable with success probability $1/3$. Therefore, using concentration of geometric random variables (Lemma A.2), we get that

[TABLE]

Noting that $K\geq 6m+24\log(1/\beta)$, this shows that the algorithm does not reach the switching budget with probability $1-O(\beta)$. Thus, the guarantees of the sparse-vector algorithm imply that the algorithm pays regret at most $\bar{\Delta}$ for each switch, hence the total regret of the algorithm is at most $O(\bar{\Delta}(m+\log(1/\beta)))=O(\bar{\Delta}\log(d/\beta))$. The claim follows as $\bar{\Delta}\leq L^{\star}+4/\eta+16B/\varepsilon$. ∎

3.1 Adaptive algorithms for DP experts

While Algorithm 1 achieves near-optimal loss for settings with low-loss experts, it requires the knowledge of the value of $L^{\star}$. As $L^{\star}$ is not always available in practice, our goal in this section is to develop an adaptive version of Algorithm 1 which obtains similar regret without requiring the knowledge of $L^{\star}$. Similarly to other online learning problems, we propose to use the doubling trick [KV05] to design our adaptive algorithms. We begin with an estimate $L^{\star}_{1}=1$ of $L^{\star}$. Then we apply Algorithm 1 using $L^{\star}=L^{\star}_{1}$ until the exponential mechanism picks an expert that contradicts the current estimate of $L^{\star}$, that is, $\sum_{i=1}^{t-1}\ell_{i}(x_{t})\gg L^{\star}_{1}$. We use the Laplace mechanism to check this privately. Noting that this happens with small probability if $L^{\star}\leq L^{\star}_{1}$, we conclude that our estimate of $L^{\star}$ was too small and set a new estimate $L^{\star}_{2}=2L^{\star}_{1}$ and repeat the same steps. As $L^{\star}\leq T$, this process will stop in at most $\log T$ phases, hence we can divide the privacy budget equally among phases while losing at most a factor of $\log T$. We present the full details in Algorithm 2.

We have the following guarantees for the adaptive algorithm.

Theorem 2**.**

Let $\ell_{1},\dots,\ell_{T}\in[0,1]^{d}$ be chosen by an oblivious adversary such that there is $x^{\star}\in[d]$ such that $\sum_{t=1}^{T}\ell_{t}(x^{\star})\leq L^{\star}$. Let $0<\beta<1/2$. Then Algorithm 2 is $\varepsilon$-DP and with probability at least $1-O(\beta)$ has regret

[TABLE]

Proof.

First we prove privacy. Note that $\bar{L}^{\star}$ can change at most $\log(T)$ times as $L^{\star}\leq T$. Therefore, we have at most $\log(T)$ applications of Algorithm 1. Each one of these is $\varepsilon/(2\log(T))$-DP. Moreover, since we have at most $K$ applications of the exponential mechanism in Algorithm 1, we have at most $K\log(T)$ applications of the Laplace mechanism in Algorithm 2. Each of these is $\varepsilon/2K\log(T)$-DP. Overall, privacy composition implies that the final privacy is $\varepsilon$-DP.

Now we prove utility. Algorithm 2 consists of at most $\log(T)$ applications of Algorithm 1 with different values of $\bar{L}^{\star}$. We will show that each of these applications incurrs low regret. Consider an application of Algorithm 1 with $\bar{L}^{\star}$. If $\bar{L}^{\star}\geq L^{\star}$, then Theorem 1 implies that the regret is at most

[TABLE]

Now consider the case where $\bar{L}^{\star}\leq L^{\star}$. We will show that Algorithm 2 will double $\bar{L}^{\star}$ and that the regret of Algorithm 1 up to that time-step is not too large. Let $t_{0}$ be the largest $t$ such that $\min_{x\in[d]}\sum_{t=1}^{t_{0}}\ell_{t}(x)\leq\bar{L}^{\star}$. Note that up to time $t_{0}$, the best expert had loss at most $\bar{L}^{\star}$ hence the regret up to time $t_{0}$ is

[TABLE]

Now let $t_{1}$ denote the next time-step when Algorithm 1 applies the exponential mechanism. Sparse-vector guarantees that in the range $[t_{0},t_{1}]$ the algorithm suffers regret at most $O\left(\bar{L}^{\star}+\frac{\log(d)+\log(T/\beta)}{\varepsilon_{0}}\right)$. Moreover, the guarantees of the Laplace mechanism imply that at this time-step, $\bar{L}_{t}\geq\bar{L}^{\star}-5K\log(T/\beta)/\varepsilon_{0}$ with probability $1-\beta$, hence Algorithm 2 will double $\bar{L}^{\star}$ and run a new application of Algorithm 1. Overall, an application of Algorithm 1 with $\bar{L}^{\star}\leq L^{\star}$ results in regret $(L^{\star}+\frac{1}{\varepsilon_{0}})\cdot\mathsf{poly}(\log\frac{Td}{\beta})$ and doubles $\bar{L}^{\star}$. Finally, note that if $\bar{L}^{\star}\geq L^{\star}+5\log(T/\beta)/\varepsilon_{0}$ then with probability $1-\beta$ the algorithm will not double the value of $\bar{L}^{\star}$. As each application of  Algorithm 1 has regret

[TABLE]

and $\bar{L}^{\star}$ is bounded by $L^{\star}+5\log(T/\beta)/\varepsilon_{0}$ with high probability, this proves the claim. ∎

3.2 A binary-tree based algorithm

In this section, we present another algorithm which achieves the optimal regret for settings with zero-expert loss. Instead of using sparse-vector, this algorithm builds on the binary tree mechanism. The idea is to repetitively select $O(\mathsf{poly}(\log(dT)))$ random good experts and apply the binary tree to calculate a private version of their aggregate losses. Whenever all of the chosen experts are detected to have non-zero loss, we choose a new set of good experts. Similarly to Algorithm 1, we can show that each new phase reduces the number of good experts by a constant factor as an oblivious adversary does not know the choices of the algorithm, hence there are only $O(\mathsf{poly}(\log(dT)))$ phases.

We provide a somewhat informal description of the algorithm in Algorithm 3. This algorithm also achieves regret $O(\mathsf{poly}(\log(dT))/\varepsilon)$ in the realizable case. We do not provide a proof as it is somewhat similar to that of Theorem 1.

4 Faster rates for DP-OCO

In this section we study differentially private online convex optimization (DP-OCO) and propose new algorithms with faster rates in the realizable setting. In Section 4.1, we develop an algorithm that reduces the OCO problem to an experts problem (by discretizing the space) and then uses our procedure for experts. In Section 4.2, we show that follow-the-regularized-leader (FTRL) using the binary tree mechanism results in faster rates in the realizable setting for smooth functions.

4.1 Experts-based algorithm for DP-OCO

The algorithm in this section essentially reduces the problem of DP-OCO to DP-OPE by discretizing the space $\mathcal{X}=\{x\in\mathbb{R}^{d}:\left\|{x}\right\|_{2}\leq D\}$ into sufficiently many experts. In particular, we consider a $\rho$-net of the space $\mathcal{X}$, that is, a set $\mathcal{X}_{\mathsf{experts}}=\{x^{1},\dots,x^{M}\}\subset\mathcal{X}$ such that for all $x\in\mathcal{X}$ there is $x^{i}\in\mathcal{X}^{\rho}_{\mathsf{experts}}$ such that $\left\|{x^{i}-x}\right\|_{2}\leq\rho$. Such a set exists if $M\geq 2^{d\log(4D/\rho)}$ ([Duc19], Lemma 7.6). Given a loss function $\ell_{t}:\mathcal{X}\to\mathbb{R}$, we define the loss of expert $x^{i}$ to be $\ell_{t}(x^{i})$. Then, we run Algorithm 1 for the given DP-OPE problem. This algorithm has the following guarantees.

Theorem 3**.**

Let $\mathcal{X}=\{x\in\mathbb{R}^{d}:\left\|{x}\right\|_{2}\leq D\}$ and $\ell_{1},\dots,\ell_{T}:\mathcal{X}\to\mathbb{R}$ be non-negative, convex and $L$-Lipschitz functions chosen by an oblivious adversary. Then running Algorithm 1 over $\mathcal{X}^{\rho}_{\mathsf{experts}}$ with $\rho=1/(LT)$ is $(\varepsilon,\delta)$-DP and with probability at least $1-O(\beta)$ has regret

[TABLE]

Proof.

Let $x_{1},\dots,x_{T}$ be the experts chosen by the algorithm. First, Theorem 1 implies that this algorithm obtains the following regret with respect to the best expert

[TABLE]

where $L^{{}^{\star}}_{\mathsf{experts}}=\min_{x\in\mathcal{X}^{\rho}_{\mathsf{experts}}}\sum_{t=1}^{T}\ell_{t}(x)$. Since $\ell_{t}$ is $L$-Lipschitz for each $t\in[T]$, we obtain that

[TABLE]

Overall this gives

[TABLE]

Setting $\rho=1/(LT\varepsilon)$ proves the claim. ∎

These results demonstrates that existing algorithms which achieve normalized regret roughly $(\ifrac{\sqrt{d}}{T\varepsilon})^{1/2}$ are not optimal for the realizable setting. Moreover, in the low-dimensional regime (constant $d$), the above bound is nearly-optimal up to logarithmic factors as we have a lower bound of $\sqrt{d}/T\varepsilon$ from the stochastic setting of this problem (see discussion in the introduction).

Finally, while the algorithm we presented in Theorem 3 has exponential runtime due to discretizing the space, we note that applying Algorithm 1 over the unit ball results in similar rates and polynomial runtime. Recall that this algorithm only accesses the loss functions to sample from the exponential mechanism, and uses sparse-vector over the running loss. Both of these can be implemented in polynomial time—since the losses are convex—using standard techniques from log-concave sampling.

4.2 Binary-tree based FTRL

In this section, we consider DP-OCO with smooth loss functions and show that DP-FTRL [KMSTTX21, Algorithm 1] with modified parameters obtains improved normalized regret ${\beta D^{2}}/{T}+({\sqrt{d}}/{T\varepsilon})^{2/3}$ in the realizable setting, compared to ${LD}/{\sqrt{T}}+({\sqrt{d}}/{T\varepsilon})^{1/2}$ in the non-realizable setting.

We present the details in Algorithm 4. Appendix B.1 in [KMSTTX21] has more detailed information about the implementation of the binary tree mechanism in DP-FTRL.

We have the following guarantees for DP-FTRL in the realizable and smooth setting.

Theorem 4**.**

*Let $\mathcal{X}=\{x\in\mathbb{R}^{d}:\left\|{x}\right\|_{2}\leq D\}$ and $\ell_{1},\dots,\ell_{T}:\mathcal{X}\to\mathbb{R}$ be non-negative, convex, $L$-Lipschitz, and $\beta$-smooth functions chosen by an oblivious adversary. DP-FTRL with $\lambda=32\beta+\left(\frac{\beta}{\varepsilon^{2}}(L/D)^{2}Td\log(T)\log(1/\delta)\right)^{1/3}$ is $(\varepsilon,\delta)$-DP and generates $x_{1},\dots,x_{T}$ that has regret

[TABLE]

For the proof, we use the following property for smooth non-negative functions.

Lemma 4.1** ([Nes04]).**

Let $\ell:\mathcal{X}\to\mathbb{R}$ be non-negative and $\beta$-smooth function. Then $\left\|{\nabla\ell(x)}\right\|_{2}^{2}\leq 4\beta\ell(x)$.

Proof.

The proof follows similar arguments to the proof of Theorem 5.1 in [KMSTTX21]. Let

[TABLE]

be the iteration of DP-FTRL where $b_{t}$ is the noise added by the binary tree mechanism. Moreover, let $\hat{x}_{t+1}$ be the non-private solution, that is,

[TABLE]

Lemma C.2 in [KMSTTX21] states that $\left\|{x_{t+1}-\hat{x}_{t+1}}\right\|_{2}\leq\left\|{b_{t}}\right\|_{2}/\lambda$. Therefore, we have

[TABLE]

where the second inequality follows from the Fenchel-Young inequality. We can now upper bound the right term. Indeed, Theorem 5.2 in [Haz16] implies that FTRL has

[TABLE]

Overall we now get

[TABLE]

The binary tree mechanism also guarantees that for all $t\in[T]$,

[TABLE]

(see Appendix B.1 in [KMSTTX21]). Thus, taking expectation and setting the regularization parameter to \lambda=32\beta+\big{(}\frac{\beta}{\varepsilon^{2}}(L/D)^{2}Td\log(T)\log(1/\delta)\big{)}^{1/3}, we have

[TABLE]

∎

5 Lower bounds

In this section, we prove lower bounds for private experts in the realizable setting which show that our upper bounds are nearly-optimal up to logarithmic factors. The lower bound demonstrates that a logarithmic dependence on $d$ is necessary even in the realizable setting. Note that for DP-OCO in the realizable setting, a lower bound of $d/T\varepsilon$ for pure DP follows from known lower bounds for DP-SCO in the interpolation regime [ACCD22] using online-to-batch conversions [Haz16].

The following theorem states our lower bound for DP-OPE.

Theorem 5**.**

Let $\varepsilon\leq 1/10$ and $\delta\leq\varepsilon/d$. If $\mathcal{A}$ is $(\varepsilon,\delta)$-DP then there is an oblivious adversary such that $\min_{x\in[d]}\sum_{t=1}^{T}\ell_{t}(x)=0$ and

[TABLE]

Proof.

Let $\ell^{0}(x)=0$ for all $x$ and for $j\in[d]$ let $\ell^{j}(x)$ be the function that has $\ell^{j}(x)=0$ for $x=j$ and otherwise $\ell^{j}(x)=1$. The oblivious adversary picks one of the following $d$ sequences uniformly at random: $\mathcal{S}^{j}=(\underbrace{\ell^{0},\dots,\ell^{0}}_{T-k},\underbrace{\ell^{j},\dots,\ell^{j}}_{k})$ where $k=\frac{\log d}{2\varepsilon}$ and $j\in[d]$. Assume towards a contradiction that the algorithm obtains regret $\log(d)/(32\varepsilon)$. This implies that there exists $d/2$ sequences such that the algorithm obtains expected regret $\log(d)/(16\varepsilon)$ where the expectation is only over the randomness of the algorithm. Assume without loss of generality these sequences are $\mathcal{S}^{1},\dots,\mathcal{S}^{d/2}$. Let $B_{j}$ be the set of outputs that has low regret on $\mathcal{S}^{j}$, that is,

[TABLE]

Note that $B_{j}\cap B_{j^{\prime}}=\emptyset$ since if $x_{1:T}\in B_{j}$ then at least $3k/4=3\log(d)/(8\varepsilon)$ of the last $k$ outputs must be equal to $j$. Now Markov inequality implies that

[TABLE]

Moreover, group privacy gives

[TABLE]

where the last inequality follows since $\delta\leq\varepsilon/d$. Overall we get that

[TABLE]

which is a contradiction for $d\geq 32$. ∎

6 Conclusion

In this work, we studied differentially private online learning problems in the realizable setting, and developed algorithms with improved rates compared to the non-realizable setting. However, several questions remain open in this domain. First, our near-optimal algorithms for DP-OPE obtain $\log^{1.5}(d)/\varepsilon$ regret, whereas the lower bound we have is $\Omega(\log(d)/\varepsilon)$. Hence, perhaps there are better algorithms with tighter logarithmic factors than our sparse-vector based algorithms. Additionally, for DP-OCO, our algorithms are optimal only for low-dimensional setting, and there remains polynomial gaps in the high-dimensional setting. Finally, optimal rates for both problems (DP-OPE and DP-OCO) are still unknown in the general non-realizable setting.

Acknowledgements

This work has received support from the Israeli Science Foundation (ISF) grant no. 2549/19 and the Len Blavatnik and the Blavatnik Family foundation.

Appendix A Concentration for sums of geometric variables

In this section, we proof a concentration result for the sum of geometric random variables, which allows us to upper bound the number of switches in the sparse-vector based algorithm. We say that $Z$ is geometric random variable with success probability $p$ if $P(W=k)=(1-p)^{k-1}p$ for $k\in\{1,2,\dots\}$. To this end, we use the following Chernoff bound.

Lemma A.1** ([MU05], Ch. 4.2.1).**

Let $X=\sum_{i=1}^{n}X_{i}$ for $X_{i}\smash{\stackrel{{\scriptstyle\rm iid}}{{\sim}}}\mathsf{Ber}(p)$. Then for $\delta\in[0,1]$,

[TABLE]

The following lemma demonstrates that the sum of geometric random variables concentrates around its mean with high probability.

Lemma A.2**.**

Let $W_{1},\dots,W_{n}$ be iid geometric random variables with success probability $p$. Let $W=\sum_{i=1}^{n}W_{i}$. Then for any $k\geq n$

[TABLE]

Proof.

Notice that $W$ is distributed according to the negative binomial distribution where we can think of $W$ as the number of Bernoulli trials until we get $n$ successes. More precisely, let $\{B_{i}\}$ for $i\geq 1$ be Bernoulli random variables with probability $p$. Then the event $W>t$ has the same probability as $\sum_{i=1}^{t}B_{i}<n$. Thus we have that

[TABLE]

We can now use Chernoff inequality (Lemma A.1) to get that for $t=2n/p$

[TABLE]

This proves that

[TABLE]

The claim now follows by noticing that $\sum_{i=1}^{n}W_{i}\leq\sum_{i=1}^{k}W_{i}$ for $W_{i}$ iid geometric random variable when $k\geq n$, thus $\mathbb{P}(\sum_{i=1}^{n}W_{i}\geq 2k/p)\leq\mathbb{P}(\sum_{i=1}^{k}W_{i}\geq 2k/p)\leq\exp{\left(-k/4\right)}$

∎