Coordinated Online Learning With Applications to Learning User Preferences

TL;DR

This paper introduces COOL, a novel online multi-task learning algorithm that coordinates related task learners via convex projections, improving learning efficiency and performance in applications like user preference modeling.

Contribution

The paper proposes COOL, a new algorithm for coordinating online learners across related tasks using convex projections, with theoretical regret bounds and practical application to user preference learning.

Findings

COOL effectively coordinates multi-task online learners.

The algorithm achieves favorable regret bounds influenced by coordination parameters.

Application to Airbnb preferences demonstrates practical utility.

Abstract

We study an online multi-task learning setting, in which instances of related tasks arrive sequentially, and are handled by task-specific online learners. We consider an algorithmic framework to model the relationship of these tasks via a set of convex constraints. To exploit this relationship, we design a novel algorithm -- COOL -- for coordinating the individual online learners: Our key idea is to coordinate their parameters via weighted projections onto a convex set. By adjusting the rate and accuracy of the projection, the COOL algorithm allows for a trade-off between the benefit of coordination and the required computation/communication. We derive regret bounds for our approach and analyze how they are influenced by these trade-off factors. We apply our results on the application of learning users' preferences on the Airbnb marketplace with the goal of incentivizing users to explore under-reviewed apartments.

Equations

$$\displaystyle\boldsymbol{w}^{*}=\big{[}(\boldsymbol{w}^{*}_{1})^{\prime}\ {\cdots}(\boldsymbol{w}^{*}_{z})^{\prime}\ {\cdots}(\boldsymbol{w}^{*}_{K})^{\prime}\big{]}^{\prime}$$

$$\displaystyle\boldsymbol{w}^{t}=\big{[}(\boldsymbol{w}^{t}_{1})^{\prime}\ {\cdots}(\boldsymbol{w}^{t}_{z})^{\prime}\ {\cdots}(\boldsymbol{w}^{t}_{K})^{\prime}\big{]}^{\prime}.$$

$$S^{\ast }=\{{\mathbf{w}}^{\ast }\in S|{\mathbf{w}}_1^{\ast }=\cdots ={\mathbf{w}}_z^{\ast }=\cdots ={\mathbf{w}}_K^{\ast }\}.$$

$$S^{\ast }=\{{\mathbf{w}}^{\ast }\in S|{\mathbf{w}}_1^{\ast }[1:d^{\prime }]\cdots ={\mathbf{w}}_z^{\ast }[1:d^{\prime }]=\cdots {\mathbf{w}}_K^{\ast }[1:d^{\prime }]\}$$

$$\{{\mathbf{w}}^{\ast }\in S|{\mathbf{w}}_{i,j}^{\ast }\in [0,r],{\mathbf{w}}_{i,j}^{\ast }\le {\mathbf{w}}_{i,k}^{\ast }+{\mathbf{w}}_{k,j}^{\ast }\forall i,j,k\in [n]\}$$

$$\displaystyle Regret_{\textsc{Alg}}(T,\boldsymbol{w}^{*})=\sum_{t=1}^{T}\Big{(}l^{t}(\boldsymbol{w}^{t}_{z^{t}})-l^{t}(\boldsymbol{w}^{*}_{z^{t}})\Big{)}.$$

$$\displaystyle\widetilde{\boldsymbol{w}}^{t+1}_{z}=\boldsymbol{w}^{t}_{z}-\eta^{t}_{z}\boldsymbol{g}^{t}_{z};\ \ \boldsymbol{w}^{t+1}_{z}=\mathop{\mathrm{argmin}}\limits_{\boldsymbol{w}\in S_{z}}\big{\lVert}\boldsymbol{w}-\widetilde{\boldsymbol{w}}^{t+1}_{z}\big{\rVert}_{2}$$

$$\displaystyle Regret_{\textsc{OL}_{z}}(T,\boldsymbol{w}_{z})=\sum_{t=1}^{T}\Big{(}l^{t}(\boldsymbol{w}^{t}_{z})-l^{t}(\boldsymbol{w}_{z})\Big{)}.$$

$$Regre{t}_{\text{textsc}{OL}_z}(T)\le \frac{3}{2}\sqrt{T}\Vert S_{max}\Vert \Vert {\mathbf{g}}_{max}\Vert .$$

$$Regre{t}_{\text{textsc}IOL}(T)\le \frac{3}{2}\sqrt{TK}\Vert S_{max}\Vert \Vert {\mathbf{g}}_{max}\Vert .$$

$${\mathbf{Q}}^t=\left[\begin{array}{ccc}\sqrt{{\tau }_1^t}& & 0\\ & \ddots & \\ 0& & \sqrt{{\tau }_K^t}\end{array}\right].$$

$${\mathbf{w}}^{t+1}$$

$$\frac{1}{2\eta }{\Vert S_{max}\Vert }^2\sqrt{TK}+2\eta {\Vert {\mathbf{g}}_{max}\Vert }^2\sqrt{TK}$$

$$+\sum _{t=1}^T\text{mathds}{1}_{\{(\neg {\xi }^{t-1})\wedge ({\xi }^t)\}}\Vert S_{max}\Vert \Vert {\mathbf{g}}_{max}\Vert$$

$$+\frac{1}{\eta }\sum _{t=1}^T\text{mathds}{1}_{\{{\xi }^t\}}\left({\delta }^t+\sqrt{2{\delta }^t}(tK{)}^{1/4}\Vert S_{max}\Vert \right)$$

$$+\frac{1}{2\eta }{\Vert S_{max}\Vert }^2-2\eta {\Vert {\mathbf{g}}_{max}\Vert }^{2}K.$$

$${\xi }^t\sim Bernoulli(\alpha )\text{ with }\alpha =\frac{c_{\alpha }}{\sqrt{T}},$$

$${\delta }^t=c_{\beta }(1-\beta {)}^2\frac{\sqrt{K}}{\sqrt{t}}{\Vert S_{max}\Vert }^2$$

$$\displaystyle\quad 2\sqrt{TK}\left\lVert S_{max}\right\rVert\left\lVert\boldsymbol{g}_{max}\right\rVert\cdot\bigg{(}1+\frac{c_{\alpha}}{2\sqrt{K}}\Big{(}1-\frac{c_{\alpha}}{\sqrt{T}}\Big{)}$$

$$\displaystyle\qquad\qquad\qquad\qquad\qquad\quad+c_{\alpha}(c_{\beta}+\sqrt{2c_{\beta}})(1-\beta)\bigg{)}.$$

$$Regre{t}_{\text{textsc}CoOL}(T)\le 2\sqrt{TK}\Vert S_{max}\Vert \Vert {\mathbf{g}}_{max}\Vert .$$

$$Regre{t}_{\text{textsc}CoOL}(T)\le \frac{3}{2}\sqrt{B}\Vert S_{max}\Vert \Vert {\mathbf{g}}_{max}\Vert ,$$

$$D_{R^t}(\mathbf{a},\mathbf{b})=R^t(\mathbf{a})-R^t(\mathbf{b})-\nabla R^t(\mathbf{b})\cdot (\mathbf{a}-\mathbf{b}),$$

$$\hat{\mathbf{b}}=\underset{\mathbf{a}\in S}{\mathrm{argmin}}{D}_{R^t}(\mathbf{a},\mathbf{b})$$

$$D_{R^t}(\mathbf{u},\mathbf{b})\ge D_{R^t}(\mathbf{u},\hat{\mathbf{b}})+D_{R^t}(\hat{\mathbf{b}},\mathbf{b}).$$

$$D_{R^t}(\mathbf{a},\mathbf{b})+D_{R^t}(\mathbf{b},\mathbf{c})=D_{R^t}(\mathbf{a},\mathbf{c})+(\mathbf{a}-\mathbf{b})(\nabla R(\mathbf{c})-\nabla R(\mathbf{b}))$$

$$\boldsymbol{x}^{t}=\big{[}(\boldsymbol{x}^{t}_{1})^{\prime}\ {\cdots}(\boldsymbol{x}^{t}_{z})^{\prime}\ {\cdots}(\boldsymbol{x}^{t}_{K})^{\prime}\big{]}^{\prime}\qquad\boldsymbol{g}^{t}=\big{[}(\boldsymbol{g}^{t}_{1})^{\prime}\ {\cdots}(\boldsymbol{g}^{t}_{z})^{\prime}\ {\cdots}(\boldsymbol{g}^{t}_{K})^{\prime}\big{]}^{\prime}.$$

$$\widetilde{\boldsymbol{w}}^{t+1}=\big{[}(\boldsymbol{w}_{1})^{\prime}\ {\cdots}(\boldsymbol{w}_{z})^{\prime}\ {\cdots}(\boldsymbol{w}_{K})^{\prime}\big{]}^{\prime}.$$

$$\sum _{z=1}^K\sqrt{{\tau }_z}\le \sqrt{TK}.$$

$$\sum _{z=1}^K\sqrt{{\tau }_z}$$

$$=\sqrt{K}\sqrt{T}$$

$$=\sqrt{TK}$$

$$\sum _{t=1}^T\frac{1}{\sqrt{t}}$$

$$=1+{\left[2\sqrt{t}\right]}_1^T$$

Full text

1 Introduction

Many real-world applications involve a number of different learning tasks. Very often, these individual tasks are related, and by sharing information between these tasks, we can improve the performance of the overall learning process. For instance, wearable devices that provide personalized recommendations to users can improve their performance by leveraging the knowledge of data from other users. This idea forms the basis of multi-task learning (Caruana, 1998).

In this paper, we study multi-task learning in the framework of online regret minimization (cf. (Cesa-Bianchi & Lugosi, 2006; Shalev-Shwartz, 2011)). We investigate the problem of online learning of $K$ related tasks (or classes/types of problems) jointly. For each task $z\in[K]$, we have a separate online learner $\textsc{OL}_{z}$ tackling the instances of this task. Task instances arrive in an arbitrary, possibly adversarial, order and each time $t$ corresponds to an instance of a task $z^{t}$ which is received by learner $\textsc{OL}_{z^{t}}$. Our goal is to coordinate the individual learners to exploit the relationship of the tasks and improve the overall performance given by the sum of regrets over all $K$ learners.

1.1 Motivating applications

Personalized AI on wearable devices. Wearable devices such as Apple Watch or Fitbit, equipped with various sensors, aim to provide realtime predictions to the users, e.g. for healthcare monitoring, based on their individual activity patterns (Jin et al., 2015). In this application, a task $z$ corresponds to providing personalized predictions to a specific user, and the learner $\textsc{OL}_{z}$ corresponds to an online learning algorithm implemented on her device. The relationship of the tasks could, for instance, help to enforce some smoothness in the predictions for users with similar demographics.

Hemimetrics encoding users’ preferences. Another motivating application, which forms the basis of the experiments in this paper (cf. Section 6), is to learn preferences of a user (or cohort of users) across different choices. These choices take the form of $n$ items available in a marketplace (e.g. items could correspond to different apartments on Airbnb). Our goal is to learn the pairwise distances $D_{i,j}$ representing the private cost of a user for switching from her default choice of item $i$ to item $j$. Knowledge about this type of preferences can be used in e-commerce applications for marketing or for maximizing social welfare, e.g. by persuading users to change their decisions (Singla et al., 2016; Kamenica & Gentzkow, 2009; Singla et al., 2015). The interaction with the users takes the form of a binary query, motivated by the posted-price model in marketplaces (Abernethy et al., 2015; Singla & Krause, 2013), where users are offered a take-it-or-leave-it offer price that they can accept or reject. The goal is to learn these $K=n^{2}-n$ distances while interacting with the users sequentially, where learning each such pairwise distance $D_{i,j}$ corresponds to one task. These distances are often correlated in real-world applications, for instance, satisfying hemimetric properties, i.e. a relaxed form of a metric (Singla et al., 2016).

In this paper, we develop an algorithmic framework to model such complex dependencies among online learners and efficiently coordinate their learning process.

1.2 Our Approach

Relationship of the tasks. In multi-task learning, one of the key aspects is modeling the relationship of the tasks. A common approach is to consider a specific structure capturing this relationship and develop an appropriate algorithm exploiting this structure: examples of this approach include shared parameters among the tasks (Chapelle et al., 2010; Jin et al., 2015), shared support (Wang et al., 2016), and smoothness in the parameters (Zhou et al., 2013). In this paper, we develop an algorithmic framework to model this relationship via a set of convex constraints. Our approach captures some of the above-mentioned structures, and allows us to model more complex dependencies.

Coordination via weighted projection. Given a generic convex set capturing task relatedness/structure, a natural question is how to design efficient algorithms to coordinate the learners to exploit this structure. For this setting, we present a principled way to coordinate learners: we show that coordination can be achieved via weighted projection (with carefully chosen weights for each learner) of the current solution vectors of the learners onto the convex set defined by the structural constraints.

Sporadic and approximate coordination. For large scale applications (i.e. large $K$), coordination at every step via projection could be computationally very expensive. Furthermore, when applying our framework in a distributed setting, it is often desirable to design a communication-efficient coordination protocol. In order to make our approach applicable in these settings, we employ two algorithmic ideas: sporadic and approximate coordinations (cf. Section 3 for details). This allows us to speed up the algorithm by an order of magnitude while retaining the improvements obtained by coordination (cf. Section 5). Furthermore, these two features have privacy-preserving properties that can be exploited in the distributed setting, if privacy or data leakage across the learners is a concern (Balcan et al., 2012): for instance by using ideas of differential privacy in online learning (cf. (Jain et al., 2012)) to control the accuracy of the projection.

1.3 Main Contributions

We develop a novel algorithm – CoOL – that employs the above-mentioned ideas to coordinate the individual online learners. We consider a standard adversarial online setting (Cesa-Bianchi & Lugosi, 2006; Shalev-Shwartz, 2011) without any probabilistic assumptions on the loss functions or order of task instances. We derive regret bounds for our approach and provide insights into how the trade-off factors controlling the rate/accuracy influence these bounds. We perform extensive experiments for learning hemimetric structures encoding users’ preferences to support the conclusions of our theoretical analysis. Furthermore, we collect data via a survey study on the Airbnb marketplace and demonstrate the practical applicability of our approach through experiments on this dataset.

2 Preliminaries

We now formalize the problem addressed in this paper.

2.1 The Model and Protocol

Tasks and Learners. We consider a set of $K$ tasks (or types/classes of problems). For each task $z\in[K]$, we have a separate online learner $\textsc{OL}_{z}$ tackling the instances of this task. For each task $z$, the online learner $\textsc{OL}_{z}$ learns some model parameters denoted by a weight vector $\boldsymbol{w}_{z}\in S_{z}\subseteq\mathbb{R}^{d_{z}}$, where $S_{z}$ denotes the feasible solution space. For simplicity of notation and w.l.o.g. we assume that $d_{z}=d\ \forall z\in[K]$. Using the standard convex online learning framework (Shalev-Shwartz, 2011; Zinkevich, 2003), we assume that $S_{z}$ is a convex, non-empty, and compact set: Let $\left\lVert S_{z}\right\rVert$ denote the diameter of the solution space for task $z$ (w.r.t the Euclidean norm)111Euclidean norm is used throughout, unless specified.. We assume $\left\lVert S_{z}\right\rVert\leq\left\lVert S_{\textnormal{max}}\right\rVert$ for some constant $\left\lVert S_{\textnormal{max}}\right\rVert$.

Online protocol. We consider an online setting, where each round is indexed by time $t$. Each learner $\textsc{OL}_{z}$ maintains a weight vector $\boldsymbol{w}^{t}_{z}$. At time $t$, the environment generates an instance of task $z^{t}$, which is received and handled by online learner $\textsc{OL}_{z^{t}}$. Hence, at time $t$, the learner $\textsc{OL}_{z^{t}}$ extends its prediction $\boldsymbol{w}^{t}_{z^{t}}$, suffers a loss $l^{t}(\boldsymbol{w}^{t}_{z^{t}})$, and updates its parameters to obtain $\boldsymbol{w}^{t+1}_{z^{t}}$. All other learners $\textsc{OL}_{z}\ \forall z\neq z^{t}$ do not interact with the environment at this time, i.e. they do not update their parameters ($\boldsymbol{w}^{t+1}_{z}=\boldsymbol{w}^{t}_{z}$). Again, using the convex online learning framework (Shalev-Shwartz, 2011; Zinkevich, 2003), we consider the loss functions to be convex, i.e. $l^{t}(\boldsymbol{w}^{t}_{z^{t}})$ is convex w.r.t. the parameter $\boldsymbol{w}^{t}_{z^{t}}$ for all $t$. We further consider gradient-based learners, and assume that learner $\textsc{OL}_{z^{t}}$ at the end of round $t$ has access to the (sub-)gradient $\boldsymbol{g}^{t}_{z^{t}}$ of the loss function $l^{t}$ computed at $\boldsymbol{w}^{t}_{z^{t}}$.222This is a weaker requirement than the full-information model where we assume access to the function $l^{t}$ and discuss this point further in our experiments, cf. Section 6. For a given task $z$, we assume that the Euclidean norm of (sub-)gradients $\left\lVert\boldsymbol{g}^{t}_{z^{t}}\right\rVert$ is upper bounded by $\left\lVert\boldsymbol{g}_{z}\right\rVert$ whenever $z^{t}=z$. Furthermore, $\left\lVert\boldsymbol{g}_{z}\right\rVert\leq\left\lVert\boldsymbol{g}_{\textnormal{max}}\right\rVert$ where $\left\lVert\boldsymbol{g}_{\textnormal{max}}\right\rVert$ is a constant. We consider a standard adversarial online setting (Cesa-Bianchi & Lugosi, 2006; Shalev-Shwartz, 2011) without any probabilistic assumptions on the loss functions.

Order of task instances. We consider a general setting where the order of the task instances and the total number of instances of any given task is arbitrary. Furthermore, in our setting each time step is associated with one task instance only. This is strictly more general than the synchronized setting, in which all the task instances arrive in parallel, e.g. as required in (Dekel et al., 2007; Lugosi et al., 2009). When implementing our algorithmc ideas for distributed optimization problems (cf. (Wang et al., 2016; Dekel et al., 2012; Shamir & Srebro, 2014)), the coordinating algorithm (e.g. implemented via the master node in a cluster) can control the schedule of the tasks, and our results directly apply in these more controlled settings as well.

2.2 Relationship of the tasks

We denote the joint solution space of the $K$ tasks as $S=S_{1}\times{\cdots}\times S_{z}\times{\cdots}\times S_{K}\subseteq\mathbb{R}^{d\cdot{K}}$. Let $\boldsymbol{w}^{*}_{z}\in S_{z}$ denote a competing weight vector for task $z$ against which we compare the regret of learner $\textsc{OL}_{z}$. For instance, $\boldsymbol{w}^{*}_{z}$ could be the optimal weight vector for task $z$ in hindsight. We define a joint competing weight vector $\boldsymbol{w}^{*}\in S\subseteq\mathbb{R}^{d\cdot{K}}$ as the concatenation of the task specific weight vectors, i.e.

[TABLE]

where $(.)^{\prime}$ denotes the transposition operator and $\boldsymbol{w}^{*}_{z}\ \forall z\in[K]$ are column vectors. Similarly, we define $\boldsymbol{w}^{t}\in S$ to be the concatenation of the task specific weight vectors at time $t$, i.e.

[TABLE]

We model the relationship of the tasks by using the following structural information: The joint competing weight vector $\boldsymbol{w}^{*}$ against which the regret of all the learners is measured lies in a convex, non-empty, and closed set $S^{*}\subseteq S$ representing a restricted joint solution space, i.e. $\boldsymbol{w}^{*}\in S^{*}$. This set $S^{*}$ can be interpreted as the prior knowledge available to the algorithm that restricts the joint weight vector $\boldsymbol{w}^{*}$ to $S^{*}$ (e.g. hemimetric structure over the pairwise distances, cf. Section 1). Note that we do not require $\boldsymbol{w}^{t}\in S^{*}$ at any given time $t$. However, our approach can also be specialized to the setting with hard constraints over the tasks’ joint weight vectors (e.g. considered in (Lugosi et al., 2009)), that would require $\boldsymbol{w}^{t}\in S^{*}\ \forall t$. For this setting, we can enforce $\boldsymbol{w}^{t}\in S^{*}\ \forall t$ by coordinating at every time step $t$ by using $\xi^{t}=1\ \forall t=1$, cf. Algorithm 1.

Next, we present three examples illustrating the kind of task relationships captured by the above model.

Unrelated tasks. $S^{*}\equiv S$ models the setting where the tasks/learners are unrelated/independent.

Shared parameters. A commonly studied setting in the distributed stochastic optimization is parameter sharing. To model this setting, $S^{*}$ is given as

[TABLE]

Instead of sharing all the parameters, another common scenario in multi-task learning is to share a few parameters. For a given $d^{\prime}\leq d$, sharing $d^{\prime}$ parameters across the tasks can be modeled by as

[TABLE]

where $\boldsymbol{w}^{*}_{z}[1\!:\!d^{\prime}]$ denotes the first $d^{\prime}$ entries in $\boldsymbol{w}^{*}_{z}$.

Hemimetric structure. As discussed in Section 1, for $n$ items available in the marketplace, the $K$ tasks can be represented as $z=(i,j)\ \forall i,j\in[n],i\neq j$, where $K=n^{2}-n$. For $d=1$, the convex set representing $r$-bounded hemimetrics is given by $S^{*}=$

[TABLE]

This structure is useful to model users’ preferences (Singla et al., 2016) and considered in the experiments in Section 5 and Section 6.

Overall, the framework of modeling the relationships via a convex set representing structural constraints is very general and can capture many complex real-world dependencies among the tasks/learners.

2.3 Objective

The focus of this paper is to develop an algorithm that plays the role of central coordinator. We measure the overall performance of the algorithm by the sum of cumulative losses of the $K$ individual learners. As is common in the online regret minimization framework (cf. (Cesa-Bianchi & Lugosi, 2006; Shalev-Shwartz, 2011)), we use regret, i.e. loss of the algorithm w.r.t. the loss of a fixed competing weight vector in hindsight, as performance measure. Considering a time horizon of $T$, the regret of the coordinating algorithm Alg against any competing weight vector $\boldsymbol{w}^{*}\in S^{*}$ is

[TABLE]

This can equivalently be written as sum of the regrets of $K$ individual learners. The objective is to develop an algorithm with low regret.

3 Methodology

In this section, we develop our main algorithm CoOL. We begin with the specification of the individual learners $\textsc{OL}_{z}$, and also present a baseline algorithm IOL without coordination among learners.

3.1 Specification of Online Learner $\textsc{OL}_{z}$

In this paper, we consider the popular algorithmic framework of online convex programming (OCP) (Zinkevich, 2003) for the individual learners $\textsc{OL}_{z}\ \forall z\in[K]$, each learning the corresponding task $z$ separately. The OCP algorithm is a gradient-descent style algorithm, similar to the online-mirror descent family of algorithms (cf. (Shalev-Shwartz, 2011)), except that it performs a projection after every gradient step to maintain feasibility of the current weight vector. In order to formally describe the algorithm, let us consider a single-task setting where $z^{t}=z\ \forall t\in[T]$. The learner $\textsc{OL}_{z}$ implementing the OCP algorithm uses the learning rate $\eta^{t}_{z}$ and updates the weight vector $\boldsymbol{w}^{t}_{z}$ in each round as follows:

[TABLE]

In this single-task setting, the regret against any competing weight vector $\boldsymbol{w}_{z}\in S_{z}$ after $T$ rounds is

[TABLE]

Theorem 1 below bounds the regret of learner $\textsc{OL}_{z}$.

Theorem 1** (From (Zinkevich, 2003)).**

*Consider the single-task setting where $z^{t}=z\ \forall t\in[T]$. For learning rate $\eta^{t}_{z}=\frac{\eta}{\sqrt{t}}$ where $\eta=\frac{\left\lVert S_{max}\right\rVert}{\left\lVert\boldsymbol{g}_{max}\right\rVert}$, the regret of the learner $\textsc{OL}_{z}$ implementing the OCP algorithm is bounded as *

[TABLE]

3.2 Independent Online Learning — IOL

As a baseline, we consider the approach of independent learning, i.e. there is no coordination among the learners. To keep track of how often task $z$ has been observed until time $t$, we introduce $\tau^{t}_{z}=\sum^{t}_{s=1}\mathds{1}_{\{z^{s}=z\}}$. Each learner $\textsc{OL}_{z}$ for task $z\in K$ maintains an individual learning rate proportional to $\nicefrac{{1}}{{\sqrt{\tau^{t}_{z}}}}$ and performs one gradient update step using Equation (2) whenever $z^{t}=z$. The regret of IOL is bounded as follows.

Theorem 2**.**

*For individual learning rates $\eta^{t}_{z}=\frac{\eta}{\sqrt{\tau^{t}_{z}}}$ where $\eta=\frac{\left\lVert S_{max}\right\rVert}{\left\lVert\boldsymbol{g}_{max}\right\rVert}$, the regret of IOL is bounded as *

[TABLE]

3.3 Coordinated Online Learning — CoOL

Now, we present our methodology for coordinating these individual online learners. Our proposed algorithm – CoOL – playing the role of a central coordinator is given in Algorithm 1, and the algorithm of the individual learners $\textsc{OL}_{z}$ according to the above-mentioned specification is given in Algorithm 3. The CoOL algorithm makes use of a function AprxProj (Function 2) for computing approximate projections.

3.3.1 High-level Overview

The execution of the Algorithm 1 is defined by two parameters: (i) the sequence of coordination steps $(\xi^{t})_{t\in[T]}$ where $\xi^{t}=0$ means that no coordination happens at time $t$ and (ii) the sequence $(\delta^{t})_{t\in[T]}$ whereby $\delta^{t}$ denotes the desired accuracy of projection at time $t$. In Algorithm 1, the sequences $(\delta^{t})_{t\in[T]}$ and $(\xi^{t})_{t\in[T]}$ are given as input, however the algorithm CoOL could also set the values of $\xi^{t}$ or $\delta^{t}$ dynamically. Our methodology operates in a synchronized way, in a sense that CoOL can coordinate with the learners at any time $t$: for clarity of presentation, we provide $(\xi^{t})_{t\in[T]}$ as input to the learners $\textsc{OL}_{z}$ as presented in Algorithm 3. The communication between the CoOL algorithm and learners is represented by RECEIVE and SHARE commands, cf. Algorithm 1 and Algorithm 3. At time $t$ when $\xi^{t}=1$, the algorithm CoOL RECEIVEs (Line $4$) the current weight vectors $\boldsymbol{w}_{z}$ and counters $\tau^{t}_{z}$ from the learners. And, CoOL SHAREs (Line $8$) the updated weight vectors $\boldsymbol{w}^{t+1}_{z}$ obtained via coordination. In the rest of this section, we will discuss the key ideas used in the development of our algorithm.

3.3.2 Coordination via Weighted Projection

Our goal is to design efficient algorithms to coordinate the learners to exploit the tasks relationship modeled by the convex set $S^{*}$. The key question to address is: At time $t$, how can we aggregate the current weight vectors $\boldsymbol{w}^{t}_{z}\ \forall z\in K$ of the individual learners to exploit the structure among the tasks? As shown in the Appendix, a principled way to coordinate in this setting is via performing a weighted projection to $S^{*}$, with weights for a learner $\textsc{OL}_{z}$ being proportional to $\sqrt{\tau^{t}_{z}}$.

We define $Q^{t}$ as a square diagonal matrix of size $dK$ with each $\sqrt{\tau^{t}_{z}}$ represented $d$ times. In the one-dimensional case ($d=1$), we can write $Q^{t}$ as

[TABLE]

Using $\widetilde{\boldsymbol{w}}$ to jointly represent the current weight vectors of all the learners at time $t$ (cf. Line $5$ in Algorithm 1), we compute the new joint weight vector $\boldsymbol{w}^{t+1}$ (cf. Line $7$ in Algorithm 1) by projecting onto $S^{*}$, using the squared Mahalanobis distance, i.e.

[TABLE]

We refer to this as the weighted projection onto $S^{*}$ and note that since $S^{*}$ is convex, the projection is unique: the weighted projection is a special case of the Bregman projection333For shared parameters, the weighted projection is equivalent to the weighted average of the parameters., cf. Appendix and (Cesa-Bianchi & Lugosi, 2006; Rakhlin & Tewari, 2009). Intuitively, the weighted projection allows us to learn about tasks that have been observed infrequently, while avoiding to “unlearn” about the tasks that have been observed more frequently. Algorithm 1, when invoked with $\xi^{t}=1,\delta^{t}=0\ \forall t\in[T]$, corresponds to a variant of our algorithm that does exact/noise-free coordination at every time step. When invoked with $\xi^{t}=0\ \forall t\in[T]$, our algorithm corresponds to the IOL baseline.

3.3.3 Sporadic & Approximate Coordination

For large scale applications (i.e. large $K$ or large $d$), coordination at every step by performing projections could be computationally very expensive: a projection onto a generic convex set $S^{*}$ would require solving a quadratic program of dimension $d\cdot K$. Furthermore, it is desirable to design a communication-efficient coordination protocol when applying our framework in a distributed setting. We extend our approach with two novel algorithmic ideas: The CoOL algorithm can perform sporadic and approximate coordinations, defined by the above-mentioned sequences $(\xi^{t})_{t\in[T]}$ and $(\delta^{t})_{t\in[T]}$. Here, $\delta^{t}$ denotes the desired accuracy or the amount of noise that is allowed at time $t$, and is given as input to the function AprxProj (Function 2) for computing approximate projections. As we shall see in our experimental results, these two algorithmic ideas of sporadic and approximate coordination allows us to speed up the algorithm by an order of magnitude while retaining the improvements obtained by coordination. Furthermore, these two features have privacy-preserving properties that can be exploited in the distributed setting, if privacy or data leakage across the learners is a concern (Balcan et al., 2012). For instance by using ideas of differential privacy in online learning (cf. (Jain et al., 2012)), the algorithm can define the desired $\delta^{t}$ at time $t$ and perturb the projected solution by noise level $\delta^{t}$.

3.3.4 Remarks

We conclude the presentation of our methodology with a few remarks. We note that the CoOL algorithm bears resemblance to the adaptive gradient based algorithms, such as AdaGrad (Duchi et al., 2011). In a completely centralized setting, another way to view this online multi-task learning problem is to treat each task as representing $d$ parameters/features of the joint online learning problem with dimension $d\cdot K$. Then, at each time $t$, observing an instance of task $z^{t}$ is equivalent to receiving a sparse (sub-)gradient with only up to $d$ non-zero entries corresponding to the features of task $z^{t}$. In fact, we can formally show that for $d=1$, when the (sub-)gradients $\boldsymbol{g}^{t}_{z^{t}}\in\{-1,1\}\ \forall t\in[T]$, the learning behavior of AdaGrad (with $S^{*}$ as the feasible solution space) and CoOL (with $\xi^{t}=1,\delta^{t}=0\ \forall t\in[T]$) are equivalent. However, our methodology is more widely applicable whereby each task is being tackled by a separate online learner, and furthermore allows us to perform sporadic/approximate coordination by controlling $\xi^{t}$ and $\delta^{t}$.

4 Theoretical Guarantees

In this section, we analyze the regret bounds of the CoOL algorithm; all proofs are provided in Appendix.

4.1 Generic Bounds

We begin with a general result in Theorem 3 and then we will refine these bounds for specific settings.

Theorem 3**.**

The regret of the CoOL algorithm is bounded by $Regret_{\textsc{CoOL}}(T)\leq$

[TABLE]

Intuitively, the regret in Theorem 3 has four components. Sporadic coordination leads to R2 and allowed noise in the projection leads to R3. R1 comes from the standard regret analysis and R4 is a constant.

4.2 Sporadic/Approx. Coordination Bounds

In our experiments, cf. Section 5, we consider the setting where CoOL projects onto $S^{*}$ approximately and with a low probability, attempting to combine the benefits of the coordination with a low average computational complexity. The regret bounds for this practically useful setting are stated in Corollary 1.

Corollary 1**.**

Set $\eta=\frac{1}{2}\frac{\left\lVert S_{max}\right\rVert}{\left\lVert\boldsymbol{g}_{max}\right\rVert}$. $\forall t\in[T]$, define:

[TABLE]

*where constants $c_{\alpha}\in[0,\sqrt{T}]$, $c_{\beta}\geq 0$, and $\beta\in[0,1]$. The expected regret of CoOL (where the expectation is w.r.t. $(\xi^{t})_{t\in[T]}$) is bounded by $\mathbf{E}\left[Regret_{\textsc{CoOL}}(T)\right]\leq$ *

[TABLE]

4.3 Bounds for Specific Settings

Algorithm 1 when invoked with $\xi^{t}=1,\delta^{t}=0\ \forall t\in[T]$ does exact/noise-free coordination at every time step. For this special case, we refine the bounds of Theorem 3 in the following corollary.

Corollary 2**.**

Set $\eta=\frac{1}{2}\frac{\left\lVert S_{max}\right\rVert}{\left\lVert\boldsymbol{g}_{max}\right\rVert}$ and $\xi^{t}=1,\delta^{t}=0\ \forall t\in[T]$. Then, the regret of CoOL is bounded by

[TABLE]

A more careful analysis for this case and by using the learning rate constant $\eta=\frac{\left\lVert S_{max}\right\rVert}{\left\lVert\boldsymbol{g}_{max}\right\rVert}$ gives a tighter regret bound with a constant factor $\frac{3}{2}$ instead of $2$. This matches the regret bound of IOL stated in Theorem 2.

This corollary shows that our approach to coordinate via weighted projection using weights as in Equation (4) preserves the worst-case guarantees of the IOL baseline algorithm. We further illustrate in the experiments (cf. Figure 1(c)) that using the wrong weights (e.g. un-weighted projection) can hinder the convergence of the learners.

This corollary also reveals the worst-case nature of the regret bounds proven in Theorem 3, i.e. the proven bounds for CoOL are agnostic to the specific structure $S^{*}$ and the order of task instances. However, for some specific setting we can get better bounds for the CoOL algorithm. For instance, in the following theorem we consider a fixed $\epsilon$-insensitive loss function and a $B$-batch order of tasks, where a task instance is repeated $B$ times before choosing a new task instance.

Theorem 4**.**

Consider $d=1$ with shared parameter structure, and $\epsilon$-insensitive loss function given by $l^{t}(\boldsymbol{w}^{t}_{z^{t}})=0\textnormal{ if }\lvert\boldsymbol{w}^{t}_{z^{t}}-c^{*}\rvert\leq\epsilon$, else $l^{t}(\boldsymbol{w}^{t}_{z^{t}})=\lvert\boldsymbol{w}^{t}_{z^{t}}-c^{*}\rvert-\epsilon$, where $\epsilon>0$ and $c^{*}\in\mathbb{R}$ is a constant. For $\eta=\frac{\left\lVert S_{max}\right\rVert}{\left\lVert\boldsymbol{g}_{max}\right\rVert}$, in the $B$-batch setting with sufficiently large batch size B\geq\big{\lceil}(\frac{\left\lVert S_{max}\right\rVert}{\epsilon}+\frac{1}{2})^{2}\big{\rceil}, the regret of the CoOL algorithm is bounded by

[TABLE]

whereas the regret bound of the IOL algorithm is worse by up to a factor $K$.

5 Experimental Evaluation

Learning hemimetrics. Our simulation experiments are based on learning hemimetrics, cf. motivating applications in Section 1.1. We consider $d=1$ and model the underlying structure $S^{*}$ as a set of $r$-bounded hemimetrics, cf. Section 2.2. Similar to (Singla et al., 2016), we generated an underlying ground-truth hemimetric $\boldsymbol{w}^{*}$ from a clustered setting where the $n$ items belong to two equal-sized clusters. We define the distance $D_{i,j}=r_{in}$ if $i$ and $j$ are from the same cluster, and $D_{i,j}=r_{out}$ otherwise. In the experiments, we set $n=10$ (resulting in $K=90$), $r_{in}=1$, and $r_{out}=9$.

Loss function and gradients. For a given instance of task $z^{t}$ at time $t$, the offer is represented by the prediction $\boldsymbol{w}^{t}_{z^{t}}$ from learner $\textsc{OL}_{z^{t}}$. We consider a simple loss function given by $l^{t}(\boldsymbol{w}^{t}_{z^{t}})=\lvert\boldsymbol{w}^{t}_{z^{t}}-\boldsymbol{w}^{*}_{z^{t}}\rvert$. When the offer $\boldsymbol{w}^{t}_{z^{t}}\geq\boldsymbol{w}^{*}_{z^{t}}$, the user “accepts” and the gradient $\boldsymbol{g}^{t}_{z^{t}}=+1$; otherwise $\boldsymbol{g}^{t}_{z^{t}}=-1$. This interaction with the users is motivated by the posted-price model in marketplaces (Abernethy et al., 2015; Singla & Krause, 2013), where users are offered a take-it-or-leave-it price, which they can accept or reject.

Projection algorithm. For computing approximation projections in AprxProj (Function 2), we adapted the triangle fixing algorithm proposed for the Metric Nearness Problem in (Brickell et al., 2008). While the original algorithm was designed for performing unweighted/exact projections for metrics, we adapted it to our setiting of weighted projections to hemimetrics with additional ability to perform approximate projection controlled by the duality gap.

5.1 Results: Order of Task Instances

In this set of experiments, we consider the CoOL algorithm with exact/noise-free coordination (i.e. $\alpha=1,\beta=1$ in Corollary 1) at every time step. We compare the effect of the order and the number of different tasks instances received — the results are shown in Figure 1, averaged over 10 runs.

Random order of tasks. Task instances $z^{t}$ are chosen uniformly at random at every time step. The CoOL algorithm suffers a significantly lower regret than the IOL algorithm, benefiting from the weighted projection onto $S^{*}$. At $T=500$, the regret of CoOL is less than half of that of the IOL, cf. Figure 1(a).

Batches of tasks. In the batch setting, a task instance is chosen uniformly at random, then it is repeated five times before choosing a new task instance. The IOL algorithm suffers a lower regret compared to the above-mentioned random order because of the higher probability that certain tasks are shown a large number of times. Furthermore, the benefit of the projection onto $S^{*}$ for the CoOL algorithm is reduced, cf. Figure 1(b), showing that the benefit of coordination depends on the specific order of the task instances for a given structure.

Single-task setting. A single task $z$ is repeated in every round. In this case, the IOL algorithm and the CoOL algorithm have same regret as illustrated, cf. Figure 1(c). In order to get better understanding of using weights $Q^{t}$ for the weighted projection, we also show a variant uw-CoOL using $Q^{t}$ as identity matrix. Un-weighted projection or using the wrong weights can hinder the convergence of the learners, as shown in Figure 1(c) for this extreme case of a single-task setting.

5.2 Results: Rate/Accuracy of Coordination

Next, we compare the trade-offs of computation vs. benefits from coordination via sporadic/approximate coordination, by varying $\alpha,\beta$ in Corollary 1.

Varying the rate of coordination ($\alpha$). The regret of the CoOL algorithm monotonically increases as $\alpha$ decreases, and is equivalent to the regret of the IOL algorithm when $\alpha=0$, cf. Figure 1(d). In the range of $\alpha$ values between $1$ and $0.1$, the regret of the CoOL algorithm is relatively constant and increases strongly only as $\alpha$ approaches [math]. With $\alpha$ as low as $0.1$, the regret of the CoOL algorithm in this setting is still almost half of that of the IOL algorithm.

Varying the accuracy of coordination ($\beta$). The regret of the CoOL algorithm monotonically increases as $\beta$ decreases, and exceeds that of the IOL algorithm for values smaller than $0.65$ because of high noise in the projections, cf. Figure 1(e). In the range of $\beta$ values between $1$ and $0.85$, the regret of the CoOL algorithm is relatively constant and less than half of that of the IOL algorithm.

Runtime vs. approximate projections. As expected, the runtime of the projection monotonically decreases as $\beta$ decreases, cf. Figure 1(f). For values of $\beta$ smaller than $0.95$, the runtime of the projection is less than $10\%$ of that of the exact projection. Thus, with $\beta$ values in the range of $0.85$ to $0.95$, the CoOL algorithm achieves the best of both the worlds: the regret is significantly smaller than that of IOL, with an order of magnitude speedup in the runtime.

6 Case Study on Airbnb Marketplace

We now study the problem of learning users’ preferences on Airbnb with the goal of incentivizing users to explore under-reviewed apartments (Kamenica & Gentzkow, 2009; Singla et al., 2016).

Airbnb dataset. Using data of Airbnb apartments from insideairbnb.com (ins, ), we created a dataset of 20 apartments as follows. We chose apartments from $4$ types in the New York City: (i) based on location (Manhattan or Brooklyn) and (ii) the number of reviews (high, $\geq 20$ or low, $\leq 2$). From each type we chose 5 apartments, resulting in a total sample of $n=20$ apartments, displayed in Figure 2(a).

Survey study on MTurk platform. In order to get real-world distributions of the users’ private costs, we collected data from Amazon’s Mechanical Turk (mtu, ) as follows. Each participant was presented two randomly chosen apartments and asked to select her preferred choice (cf. Appendix for a snapshot). Participants were then asked to specify their private cost for switching their choice to the other apartment. The resulting dataset consists of tuples $((i,j),c)$, where $i$ is the preferred choice, $j$ is the suggested choice, and $c$ is the private cost of the user. In total, we got $943$ responses/tuples. The distribution of elicited costs is shown in Figure 2(b), where NA corresponds to about $20\%$ participants who were unwilling to accept any offer. In $323$ responses $i$ was a high-reviewed apartment, $j$ an under-reviewed apartment, and participants did not select NA. We use these responses in our experiments as explained below.

Utility/rewards. A time step $t$ corresponds to a tuple $((i^{t},j^{t}),c^{t})$ with task instance $z^{t}=(i^{t},j^{t})$, and we have $T=323$. Let $p^{t}$ denote the offered price by learner $\textsc{OL}^{t}_{z^{t}}$ based on the current weight vector $\boldsymbol{w}^{t}_{z^{t}}$. We model the utility and reward of the marketplace as follows. The reward at time $t$ is $(u-p^{t})$ if the offer $p^{t}$ is accepted (i.e. $p^{t}\geq c^{t}$), and otherwise zero. Here, $u$ is the utility of the marketplace for getting a review for an under-reviewed apartment, and is set to $u=40$ in our experiments based on referral discounts given by the marketplace in past. We can model the above-mentioned rewards by the following (discontinuous) loss function: $l^{t}(p^{t})=\mathds{1}_{\{p^{t}\geq c^{t}\}}\cdot(p^{t}-c^{t})+\mathds{1}_{\{p^{t}<c^{t}\}}\cdot(u-c^{t})\textnormal{ for }u\geq c^{t}$, and $l^{t}(p^{t})=0\textnormal{ for }u<c^{t}$.

Loss function and gradients. For running the experiments, we consider a simple convex loss function given by $l^{t}(p^{t})=\mathds{1}_{\{p^{t}\geq c^{t}\}}\cdot(p^{t}-c^{t})+\mathds{1}_{\{p^{t}<c^{t}\}}\cdot\frac{u}{\Delta}\cdot(c^{t}-p^{t})$ where $\frac{u}{\Delta}$ denotes the magnitude of the gradient when a user rejects the offer, where the value of parameter $\Delta$ is set to $20$ in the experiments. Using this loss function also allows us to compute the gradients from binary feedback of acceptance/rejection of the offers.

6.1 Results

We have a total of $K=n^{2}-n$ learning tasks with $n=20$ items. Similar to Section 5, we consider $d=1$ and use a hemimetric structure to model the relationship of the tasks. The results of this experiments are shown in Figure 2(c) showing the average reward per time step and a faster convergence of the CoOL algorithm compared to that of the IOL algorithm.

7 Related Work

Online/distributed multi-task learning. Multi-task learning has been increasingly studied in online and distributed settings recently. Inspired by wearable computing, a recent work by (Jin et al., 2015) studied online multi-task learning in a distributed setting. They considered a setup where tasks arrive asynchronously and the relatedness among the tasks is maintained via a correlation matrix. However, there is no theoretical analysis on the regret bounds for the proposed algorithms. (Wang et al., 2016) recently studied the multi-task learning for distributed LASSO with shared support. Their work is different from ours — we consider general convex constraints to model task relationships and consider the adversarial online regret minimization framework.

Modeling task relationships. Similar in spirit to ours, some previous work has focused on general frameworks to model task relationships. (Dekel et al., 2007) models this via a global loss function that combines the loss values of the individual tasks incurred at a given time. This global loss function is restricted to a family of absolute norms. (Lugosi et al., 2009) models the task relationships by enforcing a set of hard constraints on the joint action space of the tasks and restrict these constraints to satisfy a Markovian property for computational efficiency. One key difference compared to (Dekel et al., 2007; Lugosi et al., 2009) is that in our work, tasks are not required to be executed simultaneously at a given time, making it applicable to distributed learning of the tasks. (Abernethy et al., 2007) studies online multi-task learning in the framework of prediction with expert advice by restricting the number of “best” experts. Another line of work, complementary to ours, considers learning the task relationships jointly with learning the tasks themselves (Kang et al., 2011; Saha et al., 2011; Ciliberto et al., 2015).

Distributed optimization. Our results have some similarity with consensus problems in the distributed (stochastic) optimization literature (Boyd et al., 2011; Dekel et al., 2012; Shamir & Srebro, 2014). (Nedic & Ozdaglar, 2009; Yan et al., 2013) study the problem of distributed autonomous online learning where each learner has its own sequence of loss functions. These learners can communicate on a network with their neighbors to share their parameters. Distributed consensus problems can be viewed as distributed multi-task problems with a constraint structure of some parameters being shared among the tasks. Our approach is applicable to these problems as long as centralized coordination is possible.

8 Conclusions

We studied online multi-task learning by modeling the relationship of tasks via a set of convex constraints. To exploit this relationship, we developed a novel algorithm, CoOL, to coordinate the task-specific online learners. The key idea of our algorithm for coordination is to perform weighted projection of the current solution vectors of the learners onto a convex set. Furthermore, CoOL can perform sporadic and approximate coordinations, thereby making it suitable for real-world applications where computation complexity is a bottleneck or low-communication/privacy is important. Our theoretical analysis yields insights into how these trade-off factors influence the regret bounds. Our experimental results on Airbnb demonstrate the practical applicability of our approach.

Appendix A Outine of the Supplement

The supplement is composed of the following sections:

• •

Appendix B introduce properties of the Bregman divergence and additional notation that we require for the later proofs of the regret bounds of the IOL and CoOL algorithm.

• •

Appendix C introduces two basic propositions that we need for the proofs in Sections D and G.

• •

Appendix D provides the proof of the regret bound of the IOL algorithm of Theorem 2.

• •

Appendix E introduces several Lemmas that we require for the proof of the regret bounds of the CoOL algorithm in Appendices G and J.

• •

Appendix F gives the intuitive idea behind using weighted projection, cf. Equation (5) in Section 3.

• •

Appendix G provides the proof of the regret bound of the CoOL algorithm of Theorem 3.

• •

Appendix H provides the proofs of Corollaries 1 and 2.

• •

Appendix I provides the proof of the regret bound of Theorem 4.

• •

Appendix J provides the proof of the tighter regret bound in Corollary 2.

• •

Appendix K provides details of the user experiment with Airbnb.

Appendix B Preleminaries

B.1 Bregman Divergence

For any strictly convex function $R^{t}:\mathbb{R}^{d}\to\mathbb{R}$, the Bregman divergence $D_{R^{t}}$ between $\boldsymbol{a}$, $\boldsymbol{b}\in\mathbb{R}^{d}$ is defined as the difference between the value of $R^{t}$ at $\boldsymbol{a}$, and the first-order Taylor expansion of $R^{t}$ around $\boldsymbol{b}$ evaluated at $\boldsymbol{a}$, i.e.

[TABLE]

We use the following properties of the Bregman divergence, cf. (Rakhlin & Tewari, 2009):

• •

The Bregman divergences is non-negative.

• •

The Bregman projection

[TABLE]

onto a convex set $S$ exists and is unique.

• •

For $\widehat{\boldsymbol{b}}$ defined as in the Bregman projection above and $\boldsymbol{u}\in S$, by the generalized Pythagorean theorem, cf. (Cesa-Bianchi & Lugosi, 2006), the Bregman divergence satisfies

[TABLE]

• •

The three-point equality

[TABLE]

follows directly from the definition of the Bregman divergence.

B.2 Notation

Throughout the supplement we use $\eta^{t}_{z}=\frac{\eta}{\sqrt{\tau^{t}_{z}}}$ and $\boldsymbol{Q}^{t}$ as per Equation (4). Similar to the definition of $\boldsymbol{w}^{t}$ in Section 2.2, we also define $\boldsymbol{x}^{t}$, and $\boldsymbol{g}^{t}$ as the concatenation of the task specific feature and gradient vectors, i.e.

[TABLE]

where for all $t$, $\boldsymbol{x}^{t}$ and $\boldsymbol{g}^{t}$ are [math] in all positions that do not correspond to task $z^{t}$. We also use $\widetilde{\boldsymbol{w}}^{t+1}$ to refer to the concatenation of the updated task specific weights, before any coordination, such that

[TABLE]

where $\boldsymbol{w}_{z}=\boldsymbol{w}^{t}_{z}$ for $z\neq z^{t}$ and $\boldsymbol{w}_{z}=\widetilde{\boldsymbol{w}}^{t}_{z}$ otherwise, cf. Algorithm 1 line 5.

Appendix C Propositions

In the following we introduce two basic propositions that we need for the proofs in Appendices D and G.

Proposition 1**.**

If $\tau_{z}\in\mathbb{R}^{+}$ for all $z\in\{1\dots K\}$, and $\sum\limits^{K}_{z=1}\tau_{z}=T,$ then

[TABLE]

Proof.

Extending and applying the Cauchy-Schwarz inequality, we get

[TABLE]

∎

Proposition 2**.**

The sum from $\sum^{T}_{t=1}\frac{1}{\sqrt{t}}$ is bounded by $2\sqrt{T}-1$.

Proof.

[TABLE]

∎

Appendix D Proof of Theorem 2

The regret in Equation (1), for any $\boldsymbol{u}\in S^{*}$ can equivalently be written as sum of the regrets of individual learners, such that

[TABLE]

where $\mathds{1}_{\{z^{t}=z\}}$ is an indicator function to denote the task at time $t$.

Proof of Theorem 2.

Applying Theorem 1 per individual learner $\textsc{OL}_{z}\ \forall z\in[K]$, we can state the regret of the algorithm IOL using Equation (6) as follows:

[TABLE]

where the last inequality follows from the Proposition 1. ∎

Appendix E Lemmas

In this Section we introduce the lemmas that we require for the proof of the regret bounds of the CoOL algorithm in Appendices G and J. Applying Lemma 1 allows us to replace the loss function with its linearization, similar to (Zinkevich, 2003). Lemmas 2 and 3 allow us to get an equivalent update procedure, using the Bregman divergence, and Lemma 4 gives a handle on the linearized regret bound, cf. (Rakhlin & Tewari, 2009). Lemma 5 uses the duality gap to upper bound the Bregman divergence between the exact and approximate projection. Lemmas 6 and 7 provide different upper bounds on the Bregman divergence.

Lemma 1**.**

For all $t$ and $\boldsymbol{w}^{t}_{z}$ there exists a $\boldsymbol{g}^{t}_{z}\in\mathbb{R}^{d}$ such that $l^{t}(\boldsymbol{w}^{t}_{z})$ can be replaced with $\boldsymbol{g}^{t}\cdot\boldsymbol{w}^{t}_{z}$ without loss of generality.

Proof.

The loss function affects the regret in two ways: first, the loss function’s gradient is used in the update step, and second, the loss function is used to calculate the regret of the algorithm. Let $\boldsymbol{g}^{t}_{z}=\nabla l^{t}(\boldsymbol{w}^{t}_{z})$ and consider the linearized loss $\boldsymbol{g}^{t}_{z}\cdot\boldsymbol{w}^{t}_{z}$. Using the linearized loss, the behavior of the algorithm remains unchanged, since $\nabla l^{t}(\boldsymbol{w}^{t}_{z})=\boldsymbol{g}^{t}_{z}$. Further, the regret either increases or remains unchanged, since the loss function is convex, such that for all $\boldsymbol{u}_{z}\in S_{z}$

[TABLE]

Rearranging, we get

[TABLE]

such that using a linearized loss, the regret either remains constant or increases. ∎

Lemma 2**.**

For $R^{t}(\boldsymbol{w})=\frac{1}{2}\boldsymbol{w}^{\prime}\boldsymbol{Q}^{t}\boldsymbol{w}$, the update rule

[TABLE]

is equivalent to the update rule

[TABLE]

Proof.

For the second update rule, inserting $R^{t}(\boldsymbol{w})=\frac{1}{2}\boldsymbol{w}^{\prime}\boldsymbol{Q}^{t}\boldsymbol{w}$ into the definition of the Bregman divergence and setting the derivative with respect to $\boldsymbol{w}$ evaluated at $\widetilde{\boldsymbol{w}}^{t+1}$ to zero, we have

[TABLE]

Rewriting, using that $\boldsymbol{g}^{t}$ is non-zero only in entries that correspond to $z^{t}$, and applying the definitions of $\boldsymbol{Q}^{t}$ and $\eta$, we get

[TABLE]

∎

Lemma 3**.**

For $R^{t}(\boldsymbol{w})=\frac{1}{2}\boldsymbol{w}^{\prime}\boldsymbol{Q}^{t}\boldsymbol{w}$, the update rule

[TABLE]

where $\widetilde{\boldsymbol{w}}^{t+1}=\boldsymbol{w}^{t}-\eta^{t}_{z}\boldsymbol{g}^{t}$, is equivalent to the update rule

[TABLE]

Proof.

Applying the definition of $R^{t}(\boldsymbol{w})$, we can rewrite

[TABLE]

∎

Lemma 4**.**

If $\boldsymbol{w}^{t+1}$ is the constraint minimizer of the objective $\eta\boldsymbol{g}^{t}\cdot\boldsymbol{w}+D_{R^{t}}(\boldsymbol{w},\boldsymbol{w}^{t})$ as stated in Lemma 3, then for any a in the solution space,

[TABLE]

Proof.

Since $\boldsymbol{w}^{t+1}$ is the constraint minimizer of the objective $\eta\boldsymbol{g}^{t}\cdot\boldsymbol{w}+D_{R^{t}}(\boldsymbol{w},\boldsymbol{w}^{t})$, any vector pointing away from $\boldsymbol{w}^{t+1}$ into the solution space has a positive product with the gradient of the objective at $\boldsymbol{w}^{t+1}$, such that

[TABLE]

Rewriting and using the three-point equality, we get

[TABLE]

∎

Lemma 5**.**

If $\widehat{\boldsymbol{w}}^{t+1}$ is the exact solution of

[TABLE]

and $\boldsymbol{w}^{t+1}\in S^{*}$ is an approximate solution with duality gap less than $\delta^{t}$, then

[TABLE]

Proof.

The duality gap is defined as the difference between the primal and dual value of the solution. The dual value is upper bounded by the optimal solution and thus less than or equal to $D_{R^{t}}(\widehat{\boldsymbol{w}}^{t+1},\widetilde{\boldsymbol{w}}^{t+1})$. Thus, for the primal solution $D_{R^{t}}(\boldsymbol{w}^{t+1},\widetilde{\boldsymbol{w}}^{t+1})$ with duality gap less than $\delta^{t}$, we have

[TABLE]

Note that $\widehat{\boldsymbol{w}}^{t+1}$ is the projection of $\widetilde{\boldsymbol{w}}^{t+1}$ onto $S^{*}$ and $\boldsymbol{w}^{t+1}\in S^{*}$. Thus, using the propertiesof the Bregman divergence we can apply the generalized Pythagorean theorem such that

[TABLE]

Inserting into the above inequality we get the result. ∎

Lemma 6**.**

For $R^{t}(\boldsymbol{w})=\frac{1}{2}\boldsymbol{w}^{\prime}\boldsymbol{Q}^{t}\boldsymbol{w}$ and $\boldsymbol{a}$ and $\boldsymbol{b}\in S$,

[TABLE]

Proof.

Using the definition of $\boldsymbol{Q}^{t}$, noting that $\left\lVert\boldsymbol{a}_{z}-\boldsymbol{b}_{z}\right\rVert^{2}\leq\left\lVert S_{max}\right\rVert^{2}$, and applying Proposition 1 we can write

[TABLE]

∎

Lemma 7**.**

For any two $\boldsymbol{a}^{t}$, $\boldsymbol{b}^{t}\in S$,

[TABLE]

Proof.

Applying our definition of $R^{t}$, we can rewrite

[TABLE]

Note that

[TABLE]

Applying Proposition 1 and using $\left\lVert\boldsymbol{a}_{z}-\boldsymbol{b}_{z}\right\rVert^{2}\leq\left\lVert S_{z}\right\rVert^{2}\leq\left\lVert S_{max}\right\rVert^{2}$ we get

[TABLE]

∎

Appendix F Idea of Weighted Projection for CoOL

The update in Algorithm 3 line LABEL:alg3.update can be equivalently written as

[TABLE]

As shown in Lemma 2, we can rewrite this as

[TABLE]

using the regularizer $R^{t}(\boldsymbol{w})=\frac{1}{2}\boldsymbol{w}\cdot\boldsymbol{Q}^{t}\boldsymbol{w}$.

Intuitively, the central coordinator CoOL restricts the solution to $S^{*}$, such that the update after coordination can be rewritten as

[TABLE]

which is equal to the weighted projection introduced in Equation (5). Using weights defined by some other heuristics could in general lead to a higher regret. For instance, in Figure 1(c) we show the increase in regret of the CoOL algorithm, when setting $\boldsymbol{Q}^{t}$ as the identity matrix.

Appendix G Proof of Theorem 3

In the following we provide the proof of Theorem 3, using notation and results of the earlier sections of the supplement. Unlike earlier work (e.g. (Zinkevich, 2003; Rakhlin & Tewari, 2009)), in our setting projections are allowed to be noisy and therefore, the solution may not be a constraint minimizer of the projection. Additionally, in our setting coordination may occur only sporadically, and thus intermediary solutions may not be in $S^{*}$. To keep track of whether coordination occurred, we define indicator functions and handle the special case of coordination at time $t$ without coordination at time $t-1$ separately.

Proof.

Proof of Theorem 3

Preparation

We define $\widehat{\boldsymbol{w}}^{t}$ as the exact solution of the projection onto $S^{*}$, such that

[TABLE]

Recall that $\xi^{t}$ is $1$ with probability $\alpha$ and [math] with probability $(1-\alpha)$. The algorithm projects onto $S^{*}$ if $\xi^{t}=1$ and onto $S_{z}$ if $\xi^{t}=0$. We define the indicator functions

[TABLE]

and the inverse

[TABLE]

as well as

[TABLE]

and the inverse

[TABLE]

Our goal is to upper bound the regret, which, using Lemma 1, we can write as

[TABLE]

Using the definitions above, we rewrite

[TABLE]

and further upper bound each sum individually.

Throughout the proof, we use the Bregman divergence with the regularizer $R^{t}(\boldsymbol{w})=\frac{1}{2}\boldsymbol{w}^{\prime}\boldsymbol{Q}^{t}\boldsymbol{w}$, and apply Lemmas 2 and 3 to get an equivalent update procedure.

Step 1: First sum

Applying Lemma 4 with $\widehat{\boldsymbol{w}}^{t+1}$ as the constraint minimizer of the objective $\eta\boldsymbol{g}^{t}\cdot\boldsymbol{w}+D_{R^{t}}(\boldsymbol{w},\boldsymbol{w}^{t})$ and $\boldsymbol{u}\in S^{*}$, we have

[TABLE]

Adding over time,

[TABLE]

In the following we upper bound each term individually. For now we leave the first term unchanged and provide an upper bound in step 3 by combining it with the results of step 2.

For the second term, we use that for our choice of $R$, the square root of the Bregman divergence is a norm and therefore satisfies the triangle inequality. Thus,

[TABLE]

Squaring both sides, we have

[TABLE]

Applying Lemmas 5 and 6, we get

[TABLE]

For the third term, using that $\boldsymbol{Q}^{1}$ is 1 in exactly one position, we have

[TABLE]

Combining and dividing by $\eta$, we get the upper bound for the first sum

[TABLE]

Step 2: Second sum

Similar to step 1, we get

[TABLE]

As in step 1, we leave the first term unchanged. For the second term, note that $w^{t+1}$ is not project onto $S^{*}$, and thus $\widehat{\boldsymbol{w}}^{t}=\boldsymbol{w}^{t}$ for all $t$, such that

[TABLE]

For the third term, similar to step 1, we have

[TABLE]

Combining, we get the upper bound for the second sum

[TABLE]

Step 3: Combination of steps 1 and 2

Note that $\mathds{1}_{\{\xi^{t}\}}+\mathds{1}_{\{\neg\xi^{t}\}}=1$ for all t. Thus, the first terms of step 1 and 2 sum to

[TABLE]

Using Lemma 7, we get

[TABLE]

Summing the remaining terms and again noting that $\mathds{1}_{\{\xi^{t}\}}+\mathds{1}_{\{\neg\xi^{t}\}}=1$, we get the upper bound for the first and second sum

[TABLE]

Step 4: Third sum

To upper bound $\sum^{T}_{t=1}\mathds{1}_{\{(\xi^{t-1})\vee(\neg\xi^{t})\}}\boldsymbol{g}^{t}\cdot(\boldsymbol{w}^{t}-\widehat{\boldsymbol{w}}^{t+1})$ we start by using Hölder’s inequality (see for example (Beckenbach & Bellman, 2012)) to get

[TABLE]

where

[TABLE]

For the norm $\left\lVert\boldsymbol{w}^{t}-\widehat{\boldsymbol{w}}^{t+1}\right\rVert_{\boldsymbol{Q}^{t}}$ we applya Lemma 4 with $\widehat{\boldsymbol{w}}^{t+1}$ as the constraint minimizer of the objective $\eta\boldsymbol{g}^{t}\cdot\boldsymbol{w}+D_{R^{t}}(\boldsymbol{w},\boldsymbol{w}^{t})$ with $\boldsymbol{w}^{t}\in S^{*}$. Using the symmetry of the Bregman divergence for our choice of $R^{t}$,

[TABLE]

and thus

[TABLE]

Note that $D_{R^{t}}(\boldsymbol{w}^{t},\widehat{\boldsymbol{w}}^{t+1})=\frac{1}{2}\left\lVert\boldsymbol{w}^{t}-\widehat{\boldsymbol{w}}^{t+1}\right\rVert^{2}_{\boldsymbol{Q}^{t}}$ and thus,

[TABLE]

Using Hölder’s inequality on the right side of the inequality, we get

[TABLE]

Therefore,

[TABLE]

We now apply the definition of the dual norm to rewrite $\left\lVert\boldsymbol{g}^{t}\right\rVert^{*}_{\boldsymbol{Q}^{t}}$. Note that $\boldsymbol{g}^{t}$ is non-zero only in position $z^{t}$ and thus

[TABLE]

The maximum is achieved at $\left\lVert\boldsymbol{x}_{z^{t}}\right\rVert_{2}=\frac{1}{\sqrt{\tau^{t}_{z^{t}}}^{1/2}}$. Thus,

[TABLE]

Inserting, summing, and using Propositions 1 and 2, we get the upper bound for the third sum,

[TABLE]

Step 5: Fourth sum

For the fourth sum we use that

[TABLE]

Using the Cauchy-Schwarz inequality, we get

[TABLE]

Thus,

[TABLE]

Step 6: Combination

Adding the results from steps 1 to 5, we get the result

[TABLE]

∎

Appendix H Proof of Corollaries

By plugging in specific algorithmic parameters into Theorem 3 we can get more concrete regret bounds on the CoOL algorithm. In the two Corollaries 1 and 2 we provide no-regret bounds for two common parametric choices, and note that similar no-regret bounds can also be achieved for different parameters.

H.1 Proof of Corollary 1

Proof of Corollary 1.

Inserting $\eta=\frac{1}{2}\frac{\left\lVert S_{max}\right\rVert}{\left\lVert\boldsymbol{g}_{max}\right\rVert}$ into the results of Theorem 3, taking the expected value over $\xi^{t}$, and using that $K\geq 1$, we get

[TABLE]

For the third term, sing $\delta^{t}=c_{\beta}(1-\beta)^{2}\frac{\sqrt{K}}{\sqrt{t}}\left\lVert S_{max}\right\rVert^{2}$ where $c_{\beta}\geq 0$, $\beta\in[0,1]$, then

[TABLE]

and, using Proposition 2 for the sum,

[TABLE]

Inserting $\alpha=\frac{c_{\alpha}}{\sqrt{T}}$ where $c_{\alpha}\in[0,\sqrt{T}]$, we get

[TABLE]

∎

H.2 Proof of Corollary 2

Proof of Corollary 2.

Inserting $\eta=\frac{1}{2}\frac{\left\lVert S_{max}\right\rVert}{\left\lVert\boldsymbol{g}_{max}\right\rVert}$ into the results of Theorem 3, we note that the first and fourth term are identical to the proof of Corollary 1.

For $\xi^{t}=1$ and $\delta^{t}=0\ \forall t\in[T]$, the second and third term equals zero. Thus, we get

[TABLE]

A more careful analysis yields the tighter regret bound

[TABLE]

The proof for the tighter bound is a bit more involved and provided in Appendix J.

∎

Appendix I Proof of Theorem 4

In this section, we provide the proof of Theorem 4, showing the improved bounds of the CoOL algorithm in a simple $B$-batch setting, in which a task instance is repeated $B$ times before choosing a new one. In the setting considered in this theorem, we have $d=1$ with shared parameter structure (cf. Section 2.2) and a $\epsilon$-insensitive loss function given by $l^{t}(\boldsymbol{w}^{t}_{z^{t}})=0\textnormal{ if }\lvert\boldsymbol{w}^{t}_{z^{t}}-c^{*}\rvert\leq\epsilon$, else $l^{t}(\boldsymbol{w}^{t}_{z^{t}})=\lvert\boldsymbol{w}^{t}_{z^{t}}-c^{*}\rvert-\epsilon$, where $\epsilon>0$ and $c^{*}\in\mathbb{R}$ is a constant.

Proof of Theorem 4.

We denote the task observed in the first batch by $z$, such that $z^{t}=z$ for $t\in 1\ldots B$.

Step 1

In the $B$-batch setting, the learner $\textsc{OL}_{z}$, corresponding to the first task receives the first $B$ task instances. Our key observation is that at the end of this batch, after $B$ time steps, where B\geq\big{\lceil}(\frac{\left\lVert S_{max}\right\rVert}{\epsilon}+\frac{1}{2})^{2}\big{\rceil}, the weight vector of this learner satisfies the condition $\lvert\boldsymbol{w}^{t}_{z}-c^{*}\rvert\leq\epsilon$.

First, using $\eta=\frac{\left\lVert S_{max}\right\rVert}{\left\lVert\boldsymbol{g}_{max}\right\rVert}$ and $\left\lVert\boldsymbol{g}_{max}\right\rVert=\left\lVert\boldsymbol{g}^{t}_{z^{t}}\right\rVert$ for all $t$, the gradient step of $\textsc{OL}_{z}$ at time $t$ is of size $\frac{\left\lVert S_{max}\right\rVert}{\sqrt{t}}$. Thus, for the gradient step to be smaller than $\epsilon$, we require $\frac{\left\lVert S_{max}\right\rVert}{\sqrt{t}}\leq\epsilon.$ Rearranging, and denoting the resulting task instance as $X$, we get

[TABLE]

Second, note that for gradient steps less than $\epsilon$, the algorithm is guaranteed to converge once the sum of gradient steps is larger than $\left\lVert S_{max}\right\rVert$. Formally, for convergence at task instance $Y$, we require

[TABLE]

or equivalently

[TABLE]

Rewriting the left side of the inequality, we get

[TABLE]

Inserting our result for $X$, and rewriting, we get

[TABLE]

which is satisfied for the setting considered in this theorem, and thus after $B$ instances, $\boldsymbol{w}^{B}_{z}$ is guaranteed to satisfy $\lvert\boldsymbol{w}^{B}_{z}-c^{*}\rvert\leq\epsilon$.

Step 2

After the end of the first batch, in the shared parameter setting weights across all tasks are equivalent after projection. Since for any solution $\boldsymbol{w}_{z}$ that satisfies $\lvert\boldsymbol{w}_{z}-c^{*}\rvert\leq\epsilon$ the gradient is zero, after $B$ instances learners do not divert from their solution and suffer zero loss for task instances $t>B$. Thus, using Corollary 2, the loss of the CoOL algorithm in this setting is bounded by

[TABLE]

Step 3

In the analysis of the IOL algorithm for this setting, weights of tasks are not shared and thus, after receiving $B$ task instances, every learner suffers the regret derived above. Thus, for $K$ learners, the IOL algorithm achieves a regret bound that is worse by up to a factor $K$. ∎

Appendix J Tighter bound for Corollary 2

Proof of Corollary 2.

Preparation

Define $\widehat{\boldsymbol{w}}^{t}$ as in the proof of Theorem 3. Note that for $\xi^{t}=1,\delta^{t}=0\ \forall t\in[T]$, $\boldsymbol{w}^{t}$ is the exact projection on $S^{*}$ for all $t$, and thus also $\widehat{\boldsymbol{w}}^{t}=\boldsymbol{w}^{t}$ for all $t$.

Our goal is to upper bound the regret

[TABLE]

We rewrite

[TABLE]

and upper bound both sums individually.

Throughout the proof we will use the Bregman divergence with the regularizer $R^{t}(\boldsymbol{w})=\frac{1}{2}\boldsymbol{w}^{\prime}\boldsymbol{Q}^{t}\boldsymbol{w}$ and apply Lemmas 2 and 3 to get an equivalent update procedure.

Step 1: First sum

For the first part of the sum, applying Lemma 4 with $\boldsymbol{w}^{t+1}$ as the constraint minimizer of the objective $\eta\boldsymbol{g}^{t}\cdot\boldsymbol{w}+D_{R^{t}}(\boldsymbol{w},\boldsymbol{w}^{t})$ and $\boldsymbol{u}\in S^{*}$, we have

[TABLE]

Adding over time,

[TABLE]

We now rewrite each term on the right side of the inequality. For the first two terms, using Lemma 7, we get

[TABLE]

For the third term, we start by applying Lemma 4 with $\boldsymbol{w}^{t+1}$ as the constraint minimizer of the objective $\eta\boldsymbol{g}^{t}\cdot\boldsymbol{w}+D_{R^{t}}(\boldsymbol{w},\boldsymbol{w}^{t})$ and $\boldsymbol{w}^{t}\in S^{*}$, such that

[TABLE]

and thus

[TABLE]

To make the inequality an equality we subtract $\Delta^{t}_{1}\geq 0$ from the right side, such that

[TABLE]

For the fourth term, using that $\boldsymbol{Q}^{1}$ is 1 in exactly one position, we have

[TABLE]

Dividing by $\eta$ and adding the second part of the sum, we get

[TABLE]

Step 2: Second sum

To bound $\frac{1}{2}\sum^{T}_{t=1}\boldsymbol{g}^{t}\cdot(\boldsymbol{w}^{t}-\boldsymbol{w}^{t+1})$ we start by using Hölder’s inequality (see for example (Beckenbach & Bellman, 2012)) to get

[TABLE]

subtracting $\Delta^{t}_{2}\geq 0$ from the right side to maintain equality,

[TABLE]

and

[TABLE]

We again use

[TABLE]

Note that $D_{R^{t}}(\boldsymbol{w}^{t},\boldsymbol{w}^{t+1})=\frac{1}{2}\left\lVert\boldsymbol{w}^{t}-\boldsymbol{w}^{t+1}\right\rVert^{2}_{\boldsymbol{Q}^{t}}$. Thus,

[TABLE]

[TABLE]

Using Hölder’s inequality and again subtracting $\Delta^{t}_{2}\geq 0$ to maintain equality,

[TABLE]

where

[TABLE]

Dividing by $\left\lVert\boldsymbol{w}^{t}-\boldsymbol{w}^{t+1}\right\rVert_{\boldsymbol{Q}^{t}}$ we get

[TABLE]

Therefore,

[TABLE]

For the first term, similar to step 4 in the proof of Theorem 3, summing and using

[TABLE]

we get

[TABLE]

Thus,

[TABLE]

Step 3: Combination

We first show that the sum of terms involving $\Delta^{t}_{1}$ and $\Delta^{t}_{2}$ is non-positive and can thus be upper bounded by [math]. Note that

[TABLE]

and

[TABLE]

Inserting and canceling identical terms,

[TABLE]

where we used Hölder’s inequality in the last step to get

[TABLE]

Inserting the remaining terms, we get

[TABLE]

Using $\eta=\frac{\left\lVert S_{max}\right\rVert}{\left\lVert\boldsymbol{g}_{max}\right\rVert}$ and $K\geq 1$,

[TABLE]

∎

Appendix K Details of the Survey Study

We recruited workers from the MTurk platform (mtu, ) to participate in the survey study. After several introductory questions about their preferences and familiarity with travel accommodations, participants were presented two randomly chosen apartments from Airbnb, using data from insideairbnb.com (ins, ). To choose between the apartment, participants were given the price, location, picture, number of reviews and rating of each apartment, as shown in Figure 3.

After the participants decided on their preference between the two randomly chosen apartments, they were told that the rental site would like to offer a special discount for the other apartment, which would reduce the price per night of that apartment. They were then asked to select the discount per night that they would like to receive to choose this apartment instead of their initial choice. The options for the answer of this questions were 0, 10, 20, 30, 40, more than 40, and NA, where participants were asked to select NA if they were not willing to consider the offer for any price. In total, we got $943$ responses, which are summarized in Figure 2(b) in Section 6.