The Bayesian Prophet: A Low-Regret Framework for Online Decision Making

TL;DR

This paper introduces a Bayesian framework for online decision-making that leverages statistical oracles, developing a novel technique called compensated coupling to establish constant regret bounds for a broad class of problems.

Contribution

The paper presents a new compensated coupling technique and demonstrates that the Bayes Selector policy achieves constant regret in online allocation problems, generalizing existing results.

Findings

Compensated coupling bounds expected regret for online policies.

Bayes Selector achieves constant regret in online allocation problems.

Results unify and extend prior online packing and matching analyses.

Abstract

We develop a new framework for designing online policies given access to an oracle providing statistical information about an offline benchmark. Having access to such prediction oracles enables simple and natural Bayesian selection policies, and raises the question as to how these policies perform in different settings. Our work makes two important contributions towards this question: First, we develop a general technique we call *compensated coupling* which can be used to derive bounds on the expected regret (i.e., additive loss with respect to a benchmark) for any online policy and offline benchmark. Second, using this technique, we show that a natural greedy policy, which we call *the Bayes Selector*, has constant expected regret (i.e., independent of the number of arrivals and resource levels) for a large class of problems we refer to as Online Allocation with finite types, which includes widely-studied Online Packing and Online Matching problems. Our results generalize and simplify several existing results for Online Packing and Online Matching, and suggest a promising pathway for obtaining oracle-driven policies for other online decision-making settings.

Figures

Tables

Table 1. Table 1: Parameters for the first Online Packing instance. Coordinates ( i , j ) 𝑖 𝑗 (i,j) represent the consumption A i ​ j subscript 𝐴 𝑖 𝑗 A_{ij} .

	Type $j$
	1	2	3	4	5	6
Resource $i=1$	1	1	0	0	1	1
Resource $i=2$	0	0	1	1	1	1
$p_j$	0.2	0.2	0.2	0.2	0.1	0.1
$r_j$	10	6	10	5	9	8

Table 2. Table 2: Parameters used for the first Online Matching instance. Coordinates ( i , j ) 𝑖 𝑗 (i,j) represent the reward r i ​ j subscript 𝑟 𝑖 𝑗 r_{ij} and r i ​ j = 0 subscript 𝑟 𝑖 𝑗 0 r_{ij}=0 implies that it is not possible to match i 𝑖 i to j 𝑗 j .

	Type $j$
	1	2	3	4	5	6
Resource 1	10	6	0	0	9	8
Resource 2	0	0	5	10	20	20
$p_j$	0.2	0.2	0.2	0.2	0.1	0.1

Table 3. Table 3: Parameters used for the second Online Packing instance. Coordinates ( i , j ) 𝑖 𝑗 (i,j) represent consumption A i ​ j subscript 𝐴 𝑖 𝑗 A_{ij} .

		Type $j$
		1	2	3	4	5	6	7	8	9	10	11	12	13	14	15
Resource $i$	1	0	1	0	1	0	1	0	0	0	0	1	0	1	0	0
	2	1	1	1	0	0	1	1	1	1	0	0	0	1	1	1
	3	0	1	1	0	1	0	0	1	0	0	1	1	0	1	0
	4	0	1	0	0	0	1	1	0	1	1	1	1	1	0	0
	5	1	1	1	0	0	0	1	0	1	0	0	0	0	1	1
	6	0	1	1	0	1	0	1	0	1	0	1	1	1	1	0
	7	1	0	1	1	1	1	1	0	1	1	1	0	0	1	1
	8	1	1	1	0	0	1	0	0	1	0	0	0	1	0	0
	9	0	1	1	1	1	0	0	1	0	0	0	1	0	0	0
	10	1	1	0	1	1	0	0	0	1	0	1	1	0	0	0
	11	1	1	0	1	0	0	1	0	1	0	0	0	0	0	1
	12	0	1	1	1	0	1	1	1	1	1	0	1	0	0	1
	13	0	0	1	0	0	1	0	1	1	0	1	1	0	1	1
	14	1	0	0	0	0	1	0	1	0	0	1	1	1	0	0
	15	0	0	0	0	0	0	0	0	0	1	0	1	0	1	0
	16	1	0	1	0	0	1	0	1	0	1	1	0	0	0	1
	17	0	1	0	1	1	0	0	0	0	1	1	1	0	0	1
	18	0	0	1	1	0	1	1	0	1	1	0	0	0	0	1
	19	1	1	1	1	0	1	0	0	0	1	0	0	0	0	0
	20	0	0	0	1	0	0	1	1	1	0	0	1	1	1	1
	$p_j$	0.075	0.075	0.125	0.025	0.05	0.062	0.062	0.1	0.1	0.05	0.125	0.012	0.075	0.062	0.002
	$r_j$	7	5	16	1	1	20	10	18	7	14	17	19	14	1	2

Table 4. Table 4: Parameters used for the second Online Matching instance. Coordinates ( i , j ) 𝑖 𝑗 (i,j) represent the reward r i ​ j subscript 𝑟 𝑖 𝑗 r_{ij} and r i ​ j = 0 subscript 𝑟 𝑖 𝑗 0 r_{ij}=0 implies that it is not possible to match i 𝑖 i to j 𝑗 j .

		Type $j$
		1	2	3	4	5	6	7	8	9	10
Resource $i$	1	10	6	0	0	9	8	2	0	0	1
	2	1	0	0	0	0	0	2	0	0	8
	3	0	0	0	0	0	0	2	0	0	6
	4	0	26	0	0	1	0	3	0	0	11
	5	1	4	0	0	0	0	0	0	0	13
	6	7	4	12	11	10	12	18	2	0	0
	$p_j$	0.1	0.1	0.1	0.1	0.1	0.1	0.1	0.1	0.1	0.1

Equations

$$\begin{array}[]{rrl}P[Z(T),B]:\quad\max&r^{\prime}x&\\ \text{s.t.}&Ax&\leq\,\,B\\ &x&\leq\,\,Z(T)\\ &x&\geq\,\,0.\end{array}$$

$$V^{\text{off}}(t,s)[\omega ]:=\underset{a\in \mathcal{A}}{\max }\{\mathcal{R}(a,s,{\theta }^t)+V^{\text{off}}(t-1,\mathcal{T}(a,s,{\theta }^t))[\omega ]\},$$

$$V^{\text{on}}(t,S^t)[\omega ]:=\sum _{\tau \le t}\mathcal{R}({\pi }^{\text{on}}(\tau ,S^{\tau },{\theta }^{\tau }),S^{\tau },{\theta }^{\tau })[\omega ]$$

$$\text{textsc}Reg[\omega ]:=V^{\text{off}}(T,S^T)[\omega ]-V^{\text{on}}(T,S^T)[\omega ]$$

$$a\in {\mathrm{argmax}}_{\hat{a}\in \mathcal{A}}\left\{\mathcal{R}(\hat{a},s,{\theta }^t)+V^{\text{off}}(t-1,\mathcal{T}(\hat{a},s,{\theta }^t))[\omega ]\right\}.$$

$$V^{\text{off}}(t,s)[\omega ]=V^{\text{off}}(t-1,\mathcal{T}({\pi }^{\text{off}}(t,s,{\theta }^t)[\omega ],s,{\theta }^t))[\omega ]+\mathcal{R}({\pi }^{\text{off}}(t,s,{\theta }^t)[\omega ],s,{\theta }^t),\forall t\in [T],s\in \mathcal{S}.$$

$$\partial R(t,a,s)$$

$$\partial r(a,j)$$

$$Q(t,a,s):=\left\{\omega \in \Omega :V^{\text{off}}(t,s)[\omega ]>\mathcal{R}(a,s,{\theta }^t)+V^{\text{off}}(t-1,\mathcal{T}(a,s,{\theta }^t))[\omega ]\right\}.$$

$$\text{textsc}Reg[\omega ]=\sum _{t\in [T]}\partial R(t,{\pi }^{\text{on}}(t,S^t,{\theta }^t),S^t)[\omega ]\cdot \text{mathds}{1}_{Q(t,S^t)}[\omega ],$$

$$V^{\text{off}}(t,S^t)[\omega ]-V^{\text{off}}(t-1,S^{t-1})[\omega ]=R^{\text{on}}(t,S^t)[\omega ]+\partial R(t,{\pi }^{\text{on}}(t,S^t,{\theta }^t),S^t)[\omega ]\text{mathds}{1}_{Q(t,S^t)}[\omega ].$$

$$\text{textsc}Reg=\sum _{t\in [T]}\partial R(t,{\pi }^{\text{on}}(t,S^t,{\theta }^t),S^t)[\omega ]\cdot \text{mathds}{1}_{Q(t,S^t)}[\omega ]=\sum _{t=T-\tau +1}^T\text{mathds}{1}_{\left\{X^t>b\right\}}+(b-X^{T-\tau })\text{mathds}{1}_{\left\{1\le X^{T-\tau }<b\right\}}.$$

$$\mathbb{E}[\text{textsc}Reg]\le \underset{a\in \mathcal{A},j\in \Theta }{\max }\partial r(a,j)\cdot \sum _{t\in [T]}\mathbb{E}[\hat{q}(t,A^t,S^t)].$$

$$\mathbb{E}[\text{textsc}Reg]\le \underset{a\in \mathcal{A},j\in \Theta }{\max }\partial r(a,j)\cdot \sum _{t\in [T]}(\mathbb{E}[\hat{q}(t,A^t,S^t)]+{\Delta }^t).$$

$$\begin{array}[]{rrl}(P_{t}^{\star})\,\max&r^{\prime}x&\\ \text{s.t.}&Ax&\leq B^{t}\\ &x&\leq Z(t)\\ &x&\geq 0.\end{array}\qquad\qquad\begin{array}[]{rrl}(P_{t})\,\max&r^{\prime}x&\\ \text{s.t.}&Ax&\leq B^{t}\\ &x&\leq\mathbb{E}[Z(t)]\\ &x&\geq 0.\end{array}$$

$$\mathrm{argmin}\{{\hat{q}}_j(t,\text{accept},b),{\hat{q}}_j(t,\text{reject},b)\}$$

$$\min \{{\hat{q}}_j(t,\text{accept},b),{\hat{q}}_j(t,\text{reject},b)\}$$

$$\mathbb{P}\left[||Z(t)-\mathbb{E}(Z(t))|{|}_{\mu }\ge \frac{\mathbb{E}[Z_j(t)]}{2{\kappa }_j}\right]\le \frac{c_j}{t^2}\forall t>{\tau }_j.$$

$$\mathbb{P}[||Z(t)-\mathbb{E}(Z(t))|{|}_1>t\epsilon ]\le e^{-t{\epsilon }^2/25},\forall 0<\epsilon <1,t\ge \frac{{\epsilon }^{2}n}{20}.$$

$$P(y)\max \{r^{\prime }x:Ax\le b,0\le x\le y,y\in {\mathbb{R}}_{\ge 0}^n\}.$$

$$q_j(t,B^t)\le \mathbb{P}[{\bar{E}}_j]+{\mathbb{P}}_j[Q(t,B^t)|E_j]\le \mathbb{P}[{\bar{E}}_j]+\text{mathds}{1}_{\left\{t<{\tilde{\tau }}_j\right\}}\le f_j(t)+\text{mathds}{1}_{\left\{t<{\tau }_j\text{ or }t<{\tilde{\tau }}_j\right\}}.$$

$$\sum _{t\in [T]}q(t,B^t)\le \sum _{j\in [n]}{p}_j\left(\sum _{t\in [T]}{f}_j(t)+\max \{{\tau }_j,{\tilde{\tau }}_j\}\right)$$

$$\sum _{t\in [T]}q(t,B^t)$$

$$\le 100{\kappa }^2\sum _{j\in [n]}\frac{1}{p_j}+3n.$$

$$\mathbb{P}[||Z(t)-\nu t|{|}_{\infty }\ge {\nu }_{j}t/2{\kappa }_j]\le n{c}^{\prime }{e}^{-c_{j}t},\forall t\in [T],j\in [n].$$

$$\underset{j,k\in [n]}{\max }\underset{s\in [0,t]}{\max }\frac{{\lambda }_j(s)}{{\lambda }_k(s)}$$

$$\underset{j\in [n]}{\min }\underset{s\in [0,t]}{\min }{\lambda }_j(s)$$

$$\mathbb{P}[D\ge d]\le \sum _{t=d}^T{q}_t$$

$$\{\omega \in \Omega :D\ge d\}\subseteq \{\omega \in \Omega :\exists t\ge d,X^t=1\}.$$

$$\begin{array}[]{rrll}(P[Z,B])\quad\max&\sum_{i,j}x_{ij}r_{ij}a_{ij}&\\ \text{s.t.}&\sum_{j}x_{ij}&\leq B_{i}&\forall i\in[d]\\ &\sum_{i\in[d]}x_{ij}&\leq Z_{j}&\forall j\in[n]\\ &\mathbf{x}&\geq 0.\end{array}$$

$$\mathbb{P}\left[||Z(t)-\mathbb{E}(Z(t))|{|}_1\ge \frac{\mathbb{E}[Z_j(t)]}{|S_j|+1}\right]\le \frac{c_j}{t^2}\forall t>{\tau }_j.$$

$$\begin{array}[]{rrll}(P_{t}^{\star})\max&\lx@intercol\sum_{i\in[d],j\in[n]}x_{ij}r_{ij}a_{ij}\hfil\lx@intercol\\ \text{s.t.}&\sum_{j\in[n]}x_{ij}&\leq B_{i}^{t}&\forall i\in[d+1]\\ &\sum_{i\in[d+1]}x_{ij}&=Z_{j}(t)&\forall j\in[n]\\ &\mathbf{x}&\geq 0.\end{array}\qquad\begin{array}[]{rrll}(P_{t})\max&\lx@intercol\sum_{i\in[d],j\in[n]}x_{ij}r_{ij}a_{ij}\hfil\lx@intercol\\ \text{s.t.}&\sum_{j\in[n]}x_{ij}&\leq B_{i}^{t}&\forall i\in[d+1]\\ &\sum_{i\in[d+1]}x_{ij}&=\mathbb{E}[Z_{j}(t)]&\forall j\in[n]\\ &\mathbf{x}&\geq 0.\end{array}$$

Full text

\OneAndAHalfSpacedXI\TheoremsNumberedThrough\ECRepeatTheorems\EquationsNumberedThrough\RRHSecondLine\LRHSecondLine

\RUNAUTHOR

Alberto Vera and Siddhartha Banerjee

\RUNTITLE

The Bayesian Prophet: A Low-Regret Framework for Online Decision Making

\TITLE

The Bayesian Prophet: A Low-Regret Framework for Online Decision Making

\ARTICLEAUTHORS\AUTHOR

Alberto Vera, Siddhartha Banerjee \AFFCornell University, \EMAILaav39@cornell.edu, \EMAILsbanerjee@cornell.edu

\ABSTRACT

We develop a new framework for designing online policies given access to an oracle providing statistical information about an offline benchmark. Having access to such prediction oracles enables simple and natural Bayesian selection policies, and raises the question as to how these policies perform in different settings. Our work makes two important contributions towards this question: First, we develop a general technique we call compensated coupling which can be used to derive bounds on the expected regret (i.e., additive loss with respect to a benchmark) for any online policy and offline benchmark. Second, using this technique, we show that a natural greedy policy, which we call the Bayes Selector, has constant expected regret (i.e., independent of the number of arrivals and resource levels) for a large class of problems we refer to as Online Allocation with finite types, which includes widely-studied Online Packing and Online Matching problems. Our results generalize and simplify several existing results for Online Packing and Online Matching, and suggest a promising pathway for obtaining oracle-driven policies for other online decision-making settings.

\KEYWORDS

Stochastic Optimization, Prophet Inequalities, Approximate Dynamic Programming, Revenue Management, Online Allocation, Online Packing, Online Matching.

“Life is the sum of all your choices.”

— Albert Camus

1 Introduction

Everyday life is replete with settings where we have to make decisions while facing uncertainty over future outcomes. Some examples include allocating cloud resources, matching an empty car to a ride-sharing passenger, displaying online ads, selling airline seats, etc. In many of these instances, the underlying arrivals arise from some known generative process. Even when the underlying model is unknown, companies can turn to ever-improving machine learning tools to build predictive models based on past data. This raises a fundamental question in online decision-making: how can we use predictive models to make good decisions?

Broadly speaking, an online decision-making problem is defined by a current state and a set of actions, which together determine the next state as well as generate rewards. In Markov decision processes (MDPs), the rewards and state transitions are also affected by some random shock. Optimal policies for such problems are known only in some special cases, when the underlying problem is sufficiently simple, and knowledge of the generative model sufficiently detailed. For many problems of interest, an MDP approach is infeasible due to two reasons: $(1)$ insufficiently detailed models of the generative process of the randomness, and $(2)$ the complexity of computing the optimal policy (the so-called ‘curse of dimensionality’). These shortcomings have inspired a long line of work on approximate dynamic programming (ADP).

We focus on a general class of online resource allocation problems, which we refer to as Online Allocation (cf. Section 2.1), and which generalizes two important classes of online decision-making problems: Online Packing and Online Matching. In brief, Online Allocation problems involve a set of $d$ distinct resources, and a principal with some initial budget vector $B\in\mathbb{N}^{d}$ of these resources, which have to be allocated among $T$ incoming agents. Each agent has a type comprising of some specific requirements for resources and associated rewards. The exact type becomes known only when the agent arrives. The principal must make irrevocable accept/reject decisions to try and maximize rewards, while obeying the budget constraints.

Online Packing and Online Matching problems are fundamental in MDP theory; they have a rich existing literature and widespread applications in many domains. Nevertheless, our work develops new policies for both these problems which admit performance guarantees that are order-wise better than existing approaches. These policies can be stated in classical ADP terms (for example, see Algorithms 2 and 3), but draw inspiration from ideas in Bayesian learning. In particular, our policies can be derived from a meta-algorithm, the Bayes selector (Algorithm 1), which makes use of a black-box prediction oracle to obtain statistical information about a chosen offline benchmark, and then acts on this information to make decisions. Such policies are simple to define and implement in practice, and our work provides new tools for bounding their regret vis-á-vis the offline benchmark. Thus, we believe that though our theoretical guarantees focus on a particular class of Online Allocation problems, our approach provides a new way for designing and analyzing much more general online decision-making policies using predictive models.

1.1 Our Contributions

We believe our contributions in this work are threefold:

Technical: We present a new stochastic coupling technique, which we call the compensated coupling, for evaluating the regret of online decision-making policies vis-à-vis offline benchmarks. 2. 2.

Methodological: Inspired by ideas from Bayesian learning, we propose a class of policies, expressed as the Bayes Selector, for general online decision-making problems. 3. 3.

Algorithmic: For a wide class of problems which we refer to as Online Allocation (which includes Online Packing and Online Matching problems), we prove that the Bayes Selector gives expected regret guarantees that are independent of the size of the state-space, i.e., constant with respect to the horizon length and budgets.

Organization of the paper:

The rest of the paper is organized as follows: In Section 2, we introduce a general problem, called Online Allocation, which includes as special cases the Multi-Secretary, Online Packing and Online Matching problems, and also more general settings involving agents with complex valuations over bundles. We also define the prophet benchmark, and discuss shortcomings of prevailing approaches. In Section 3, we present our main technical tool, the Compensated Coupling, in the general context of finite-state finite-horizon MDPs; we illustrate its use by applying it to the ski-rental problem. In Section 3.3 we introduce the Bayes Selector policy, and discuss how the compensated coupling provides a generic recipe for obtaining regret bounds for such a policy. In Sections 4 and 5, we use these techniques for the Online Packing and Online Matching problems; we analyze them separately to exploit their structure and obtain stronger results. Finally, in Section 6 we analyze the most general problem (Online Allocation).

In particular, in Section 4, we propose a Bayes Selector policy for Online Packing and demonstrate the following performance guarantee:

Theorem 1.1** **(Informal)

For any Online Packing problem with a finite number of resource types and arrival types, under mild conditions on the arrival process, the regret of the Bayes Selector is independent of the horizon $T$ and budgets $B$ (in expectation and with high probability).

In more detail, our regret bounds depend on the ‘resource matrix’ $A$ and the distribution of arriving types, but are independent of $T$ and $B$. Moreover, the results holds under weak assumptions on the arrival process, including Multinomial and Poisson arrivals, time-dependent processes, and Markovian arrivals. This result generalizes prior and contemporaneous results [36, 29, 13, 41, 3]. We show similar results for Online Matching problems in Section 5.

1.2 Related Work

Our work is related to several active areas of research in MDPs and online algorithms.

Approximate Dynamic Programming:

The complexity of computing optimal MDP solutions can scale with the state space, which often makes it impractical (the so-called ‘curse of dimensionality’ [35]). This has inspired a long line of work on approximate dynamic programming (ADP) [35, 38] to develop lower complexity heuristics. Although these methods often work well in practice, they require careful choice of basis functions, and any bounds are usually in terms of quantities which are difficult to interpret. Our work provides an alternate framework, which is simpler and has interpretable guarantees.

Model Predictive Control:

A popular heuristic for ADP and control which is closer to our paradigm is that of model predictive control (MPC) [34, 7, 17]. MPC techniques have also been connected with online convex optimization (OCO)  [28, 15, 14] to show how prediction oracles can be used for OCO, and applying these policies to problems in power systems and network control. These techniques however require continuous controls, and do not handle combinatorial constraints.

Information Relaxation:

Parallel to the ADP focus on developing better heuristics, there is a line of work on deriving upper bounds via martingale duality, sometimes referred to as information relaxations [8, 19, 9]. The main idea in these works is to obtain performance bounds for heuristic policies work by defining more refined outer bounds; in particular, this can be done by adding a suitable martingale term to the current reward, in order to penalize ‘future information’. Our offline benchmarks serve a similar purpose; however, a critical difference is that instead of using these to analyze a given heuristic, we use the benchmarks directly to derive control policies.

Online Packing and Prophet Inequalities:

The majority of work focuses on competitive ratio bounds under worst-case distributions. In particular, there is an extensive literature on the so-called Prophet Inequalities, starting with the pioneering work of [27], to more recent extensions and applications to algorithmic economics [31, 23, 18, 2]. We note that any competitive ratio guarantee implies a $O(T)$ expected regret, in comparison to our $O(1)$ expected regret guarantees – the cost for this, however, is that our results hold under more restrictive assumptions on the inputs. For example, the policy suggested by [23] is static and [3, Theorem 1] shows that any static policy has $\Omega(\sqrt{T})$ expected regret, hence it cannot yield a strong guarantee like ours.

Distribution-agnostic and Adversarial Models:

Though we focus only on the case where the input is drawn from a stochastic process, we note that there is a long line of work on Online Packing with adversarial inputs [11, 12, 30], and also distribution-agnostic approaches [5, 20]. More generally, there is a large body of work on using sublinear expected regret algorithms for solving online linear and convex programs [1, 24]. The algorithms in these works are incomparable to ours in that, while they cannot get constant expected regret in our setting (stochastic input, finite type space), they provide guarantees under much weaker assumptions.

Regret bounds in Stochastic Online Packing:

For these problems, regret is the most meaningful metric to study, see [42] for a discussion, where approximations to the regret are studied. The first work to prove constant expected regret in a context similar to ours is [3], who prove a similar result for the multi-secretary setting with multinomial arrivals; we strengthen their result in Theorem 4.2. This result is relevant to a long line of work in applied probability. Some influential works are [36], which provides an asymptotically optimal policy under the diffusion scaling, and [29] who provide a resolving policy with constant expected regret under a certain non-degeneracy condition. In contrast, degeneracy plays no role in the performance of our algorithms. More recently, [13] partially extended the result of [3] for more general packing problems; their guarantee is only valid for i.i.d. Poisson arrival processes and when the system is scaled linearly, i.e., when $B$ is proportional to $T$ (our results and [3] make no such assumption). In Section 7, we demonstrate with a numerical study that the Bayes Selector far outperforms all these previous policies.

2 Problem Setting and Overview of Results

As we mentioned in the introduction, our contributions in this work are two-fold – (i) we give a technique to analyze the regret of any MDP, and (ii) we apply it to specific problems to obtain constant regret. Our focus in this work is on the subclass of Online Packing problems with stochastic inputs. This is a subclass of the wider class of finite-horizon online decision-making problems: given a time horizon $T\in\mathbb{N}$ with discrete time-slots $t=T,T-1,\ldots,1$, we need to make a decision at each time leading to some cumulative reward. Note that throughout our time index $t$ indicates the time to go. We present the details of our technical approach in this more general context whenever possible, indicating additional assumptions when required.

In what follows, we use $[k]$ to indicate the set $\{1,2,\ldots,k\}$, and denote the $(i,j)$-th entry of any given matrix $A$ interchangeably by $A_{i,j}$ or $A(i,j)$. We work in an underlying probability space $(\Omega,\mathcal{F},\mathbb{P})$, and the complement of any event $Q\subseteq\Omega$ is denoted $\bar{Q}$. For any optimization problem $(P)$, we use $v(P)$ to indicate its objective value. If $S$ is a finite set, $\lvert S\rvert$ denotes cardinality. The set $\mathbb{N}$ of naturals includes zero.

2.1 The Online Allocation Problem

We now present a generic problem, which we refer to as Online Allocation, that encompasses both Online Matching and Online Packing. The setup is as follows: There are $d$ distinct resource-types denoted by the set $[d]$, and at time $t=T$, we have an initial availability (budget) vector $B=(B_{1},B_{2},\ldots,B_{d})\in\mathbb{N}^{d}$. At every time $t=T,T-1,\ldots,1$, an arrival with type $\theta^{t}$ is drawn from a finite set of $n$ distinct types $\Theta=[n]$, via some distribution which is known to the algorithm designer (henceforth referred to as the principal). We denote $Z(t)=(Z_{1}(t),Z_{2}(t),\ldots,Z_{n}(t))\in\mathbb{N}^{n}$ as the cumulative vector of the last $t$ arrivals, where $Z_{j}(t)\coloneqq\sum_{\tau\leq t}\mathds{1}_{\left\{\theta^{\tau}=j\right\}}$.

Each arriving agent is associated with a choice-over-bundles. Formally, an arriving agent of type $j$ desires any one among a collection of multisets $S_{j}\subseteq 2^{[d]}$; the principal can allocate any $s\in S_{j}$ (referred to as a bundle) to the agent, thereby obtaining a reward $r_{sj}$ and consuming one unit of each resource $i\in s$. Observe that we do not assume additive valuations, e.g, we do not require $r_{\{1,2\}j}=r_{\{1\}j}+r_{\{2\}j}$. The model can naturally be extended to allow bundles to consume multiple units of any resource.

At each time, the principal must decide whether to allocate a bundle to the request $\theta^{t}$ (thereby generating the associated reward while consuming the required resources), or reject it (no reward and no resource consumption). Allocating a bundle requires that there is sufficient budget of each resource to cover the request. The principal’s aim is to make irrevocable decisions so as to maximize overall rewards.

In Section 6, we present results for this general problem. We next describe three particular cases of the general problem, which are each of independent interest. We analyze these three cases separately since they admit improved results over the general case.

Multi-Secretary.

This is a fundamental one dimensional instance. In this problem, we have $B\in\mathbb{N}$ available positions and want to hire employees with the highest abilities (rewards). There is one resource type $(d=1)$ with budget $B$; each employee occupies one unit of budget (one position). This is an instance of Online Allocation, with $S_{j}=\{\{1\}\}$ for all $j$ (all candidates want the same resource) and rewards $r_{\{1\}j}=r_{j}$.

Online Packing.

In this multidimensional problem, each request $j$ is associated to one bundle and one reward if allocated. Specifically, we are given a consumption matrix $A\in\mathbb{N}^{d\times n}$, where $a_{ij}$ denotes the units of resource $i$ required to serve the request $j$ and, for each $j$, there is a reward $r_{j}$ if served. This is an instance of Online Allocation, wherein agents of each type $j$ desire a single bundle: $S_{j}=\{\{a_{ij}\text{ units of }i\text{ for each resource }i\in[d]\}\}$.

Online Matching.

There are $d$ resources, but now each type $j$ wants at most one resource from among a given set of resources. Formally, a type $j$ agent wants any from $A_{j}$ instead of all from $A_{j}$. The types can be represented by a reward matrix $r\in\mathbb{R}_{\geq 0}^{d\times n}$ and adjacency matrix $A\in\{0,1\}^{d\times n}$; if the arrival is of type $j\in[n]$, we can allocate at most one resource $i$ such that $a_{ij}=1$, leading to a reward of $r_{ij}$. This problem can be thought as online bipartite matching, see Section 5 for details. It corresponds to an instance of Online Allocation, with bundles $S_{j}=\{\{i\}\text{ for each resource }i\in[d]\text{ s.t. }a_{ij}=1\}$ and rewards $r_{\{i\}j}=r_{ij}$.

2.2 Arrival Processes

To specify the generative model for the type sequence $\theta^{T},\theta^{T-1},\ldots,\theta^{1}$, an important subclass is that of stationary independent arrivals, which further admits two widely-studied cases:

$1.$ The Multinomial process is defined by a known distribution $p\in\mathbb{R}_{\geq 0}^{n}$ over $[n]$; at each time, the arrival is of type $j$ with probability $p_{j}$, thus $Z(t)\sim\operatorname{Multinomial}(t,p_{1},\ldots,p_{n})$.

$2.$ The Poisson arrival process is characterized by a known rate vector $\lambda\in\mathbb{R}_{\geq 0}^{n}$. Arrivals of each class are assumed to be independent such that $Z_{j}(t)\sim\operatorname{Poisson}(\lambda_{j}t)$. Note that, although this is a continuous-time process, it can be accommodated in a discrete-time formulation by defining as many periods as arrivals (see Section 10.1 for details).

We assume w.l.o.g. that $p_{j}>0$ and $\lambda_{j}>0$ for all $j\in[n]$ (if this is not the case for some $j$, that type never arrives and can be removed from the instance description). More general models allow for non-stationary and/or correlated arrival processes – for example, non-homogeneous Poisson processes, Markovian models (see Example 4.13), etc. An important feature of our framework is that it is capable of handling a wide variety of such processes in a unified manner, without requiring extensive information regarding the generative model. We discuss the most general assumptions we make on the arrival process in Section 4.2.

2.3 The Offline Benchmark

Suppose a given problem is simultaneously solved by two ‘agents’, Online and Offline, who are primarily differentiated based on their access to information. Online can only take non-anticipatory actions, i.e., use available information only, whereas Offline is allowed to make decisions with full knowledge of future arrivals. This is known in the literature as a prophet or full-information benchmark. Denoting the total collected rewards of Offline and Online as $V^{\textnormal{off}}$ and $V^{\textnormal{on}}$ respectively, we define the regret to be the additive loss $\textsc{Reg}\coloneqq V^{\textnormal{off}}-V^{\textnormal{on}}$. Observe that $V^{\textnormal{on}}$ depends on the policy used by Online, the underlying policy will always be clear from context. Our aim is to design policies with low $\mathbb{E}[\textsc{Reg}]$.

For Online Packing, the solution to Offline’s problem corresponds to solving an integer programming problem. A looser, but more tractable benchmark, is given by an LP relaxation of this policy: given arrivals vector $Z(T)$, we assume Offline solves the following:

[TABLE]

The unavoidable regret of the fluid benchmark

: The most common technique for obtaining policies for Online Packing is based on the so-called fluid (a.k.a. deterministic or ex ante) LP benchmark $(P[\mathbb{E}[Z(T)],B])$, where $(P)$ is defined in Eq. 1. It is easy to see via Jensen’s Inequality that $v(P[\mathbb{E}[Z(T)],B])\geq\mathbb{E}[v(P[Z(T),B])]$, and hence the fluid LP is an upper bound for any online policy. Although the use of this fluid benchmark is the prevalent tool to bound the regret in Online Packing problems [37, 36, 29, 41], the following result shows that the approach of using $v(P[\mathbb{E}[Z(T)],B])$ as a benchmark can never lead to a constant expected regret policy, as the fluid benchmark can be far off from the optimal solution in hindsight.

Proposition 2.1

For any Online Packing problem, if the arrival process satisfies the Central Limit Theorem and the fluid LP is dual degenerate, i.e., the optimal dual variables are not unique, then $v(P[\mathbb{E}[Z(T)],B])-\mathbb{E}[v(P[Z(T),B])]=\Omega(\sqrt{T})$.

This gap has been reported in literature, both informally and formally (see [3, 13]). For completeness we provide a proof in Section 9. Note though that this gap does not pose a barrier to showing constant-factor competitive ratio guarantees, i.e. $O(T)$ expected regret; the fluid LP benchmark is widely used for prophet inequalities. In contrast, the gap presents a barrier for obtaining $O(1)$ expected regret bounds. Breaking this barrier thus requires a fundamentally new approach.

2.4 Overview of our Approach and Results

Our approach can be viewed as a meta-algorithm that uses black-box prediction oracles to make decisions. The quantities estimated by the oracles are related to our offline benchmark and can be interpreted as probabilities of regretting each action in hindsight. A natural ‘Bayesian selection’ strategy given such estimators is to adopt the action that is least likely to cause regret in hindsight. This is precisely what we do in Algorithm 1, and hence, we refer to it as the Bayes Selector.

Bayesian selection techniques are often used as heuristics in practice. Our work however shows that such policies in fact have excellent performance for online allocation; in particular, we show that for matching and packing problems:

There are easy to compute estimators (in particular, ones which are based on simple adaptive LP relaxations) that, when used for Algorithm 1, give constant expected regret for a wide range of distributions (see Theorems 4.2, 4.6 and 5.1). 2. 2.

Using other types of estimators (for example, Monte Carlo estimates) in Algorithm 1 yields comparable performance guarantees (see Corollaries 4.7 and 5.2).

At the core of our analysis is a novel stochastic coupling technique for analyzing online policies based on offline (or prophet) benchmarks. Unlike traditional approaches for regret analysis which try to show that an online policy tracks a fixed offline policy, our approach is instead based on forcing Offline to follow Online’s actions. We describe this in more detail in the next section.

3 Compensated Coupling and the Bayes Selector

We introduce our two main technical ideas: $1.$ the compensated coupling technique, and $2.$ the Bayes selector heuristic for online decision-making. The techniques introduced here are valid for any generic MDP; in subsequent sections we specialize them to Online Allocation.

3.1 MDPs and Offline Benchmarks

The basic MDP setup is as follows: at each time $t=T,T-1,\ldots,1$ (where $t$ represents the time-to-go), based on previous decisions, the system state is one of a set of possible states $\mathcal{S}$. Next, nature generates an arrival $\theta^{t}\in\Theta$, following which we need to choose from a set of available actions $\mathcal{A}$. The state updates and rewards are determined via a transition function $\mathcal{T}:\mathcal{A}\times\mathcal{S}\times\Theta\to\mathcal{S}$ and a reward function $\mathcal{R}:\mathcal{A}\times\mathcal{S}\times\Theta\to\mathbb{R}$: for current state $s\in\mathcal{S}$, arrival $j\in\Theta$ and action $a\in\mathcal{A}$, we transition to the state $\mathcal{T}(a,s,j)$ and collect a reward $\mathcal{R}(a,s,j)$. Infeasible actions $a$ for a given state $s$ correspond to $\mathcal{R}(a,s,j)=-\infty$. The sets $\mathcal{A},\mathcal{S},\Theta$, as well as the measure over arrival process $\{\theta^{t}:t\in[T]\}$, are known in advance. Finally, though we focus mainly on maximizing rewards, the formalism naturally ports over to cost-minimization.

Recall that we adopt the view that the problem is simultaneously solved by two ‘agents’: Online and Offline. Online can only take non-anticipatory actions while Offline makes decisions with knowledge of future arrivals. To keep the notation simple, we restrict ourselves to deterministic policies for Offline and Online, thereby implying that the only source of randomness is due to the arrival process (our results can be extended to randomized policies).

A sample path $\omega\in\Omega$ encodes the arrival sequence $\{\theta^{t}:t\in[T]\}$. In other words, there exists a unique sequence of types that is consistent with $\omega$; whenever we fix $\omega$, the type $\theta^{t}$ is uniquely identified, but, for notational ease, we do not write $\theta^{t}[\omega]$. For a given sample-path $\omega\in\Omega$ and time $t$ to go, Offline’s value function is specified via the deterministic Bellman equations

[TABLE]

with boundary condition $V^{\textnormal{off}}(0,s)[\omega]=0$ for all $s\in\mathcal{S}$. The notation $V^{\textnormal{off}}(t,s)[\omega]$ is used to emphasize that, given sample-path $\omega$, Offline’s value function is a deterministic function of $t$ and $s$.

We require that the DP formulation in Eq. 2 is well defined. For simplicity, we enforce this with the following assumption: there are some constants $c_{1},c_{2}\geq 0$ such that $-c_{1}\leq\max_{a\in\mathcal{A}}\mathcal{R}(a,s,j)\leq c_{2}$ for all $s\in\mathcal{S},j\in\Theta$. In other words, every state has a feasible action and the maximum reward is uniformly bounded and attained. The spaces $\mathcal{S},\Theta,\mathcal{A}$ and functions $\mathcal{T},\mathcal{R}$ are otherwise arbitrary. We enforce this assumption for clarity of exposition, but we observe that it can be further generalized, c.f. [6, Volume II, Appendix A].

On the other hand, Online chooses actions based on policy $\pi^{\text{on}}$ defined as follows:

Definition 3.1** **(Online Policy)

An online policy $\pi^{\text{on}}$ is any collection of functions $\{\pi^{\text{on}}(t,s,j):t\in[T],s\in\mathcal{S},j\in\Theta\}$ such that, if at time $t$ the current state is $s$ and a type $j$ arrives, then Online chooses the action $\pi^{\text{on}}(t,s,j)\in\mathcal{A}$. The function $\pi^{\text{on}}(t,\cdot,\cdot)$ can depend only on $\{\theta^{T},\cdots,\theta^{t}\}$, i.e., on the randomness observed at periods $\tau\geq t$ (the history).

Let us denote $\{S^{t}:t\in[T]\}$ as Online’s state over time, i.e., the stochastic process $S^{t}\in\mathcal{S}$ that results from following a given policy $\pi^{\text{on}}$. We can write Online’s accrued value, for a given policy $\pi^{\text{on}}$, as

[TABLE]

For notational ease, we omit explicit indexing of $V^{\textnormal{on}}$ on policy $\pi^{\text{on}}$.

On any sample-path $\omega$, we can define the regret of an online policy to be the additive loss incurred by Online using $\pi^{\text{on}}$ w.r.t. Offline, i.e.,

[TABLE]

Remark 3.2** **(Regret is Agnostic of Offline Algorithm)

Our previous definition of Reg depends only on the online policy $\pi^{\text{on}}$, but it does not depend on the policy (or algorithm) used by Offline as long as it is optimal. For example, in the case where there are multiple maximizers in the Bellman Eq. 2, different tie-breaking rules for Offline yield different algorithms, but all of them are optimal and have the same optimal value $V^{\textnormal{off}}$.

3.2 The Compensated Coupling

At a high-level, the compensated-coupling is a sample path-wise charging scheme, wherein we try to couple the trajectory of a given policy to a sequence of offline policies. Given any non-anticipatory policy (played by Online), the technique works by making Offline follow Online – formally, we couple the actions of Offline to those of Online, while compensating Offline to preserve its collected value along every sample-path.

Example 3.3

Consider the multi-secretary problem with budget $B=1$ and three arriving types $\Theta=\{1,2,3\}$ with $r_{1}>r_{2}>r_{3}$. The state space in this problem is $\mathcal{S}=\mathbb{N}$ and the action space is $\mathcal{A}=\{\text{accept},\text{reject}\}$. Suppose for $T=4$ the arrivals on a given sample-path are $(\theta^{4},\theta^{3},\theta^{2},\theta^{1})=(1,2,1,3)$. Note that Offline will accept exactly one arrival of type $1$, but is indifferent to which arrival. While analyzing Online, we have the freedom to choose a benchmark by specifying the tie-breaking rule for Offline– for example, we can compare Online to an Offline agent who chooses to front-load the decision by accepting the arrival at $t=4$ (i.e., as early in the sequence as possible) or back-load it by accepting the arrival at $t=2$. In conclusion, for this sample path, the following two sequences of actions are optimal for Offline: (accept,reject,reject,reject) and (reject,reject,accept,reject).

Suppose instead that we choose to reject the first arrival ($t=4$), and then want Offline to accept the type-$2$ arrival at $t=3$ – this would lead to a decrease in Offline’s final reward. The crucial observation is that we can still incentivize Offline to accept arrival type $2$ by offering a compensation (i.e., additional reward) of $r_{1}-r_{2}$ for doing so. The basic idea behind the compensated coupling is to generalize this argument. We want Offline to take Online’s action, hence we couple the states of Offline and Online with the use of compensations.

We start with a general problem: Given sample-path $\omega\in\Omega$ with arrivals $\{\theta^{t}[\omega]:t\in[T]\}$, recall $V^{\textnormal{off}}(t,s)[\omega]$ denotes Offline’s value starting from state $s$ with $t$ periods to go. $V^{\textnormal{off}}(t,s)[\omega]$ obeys the Bellman Eq. 2. The following definition is about the actions satisfying said Bellman Equations.

Definition 3.4** **(Satisfying Action)

Fix $\omega\in\Omega$. For any given state $s$ and time $t$, we say Offline is satisfied with an action $a$ at $(s,t)$ if $a$ is a maximizer in the Bellman equation, i.e.,

[TABLE]

Observe that $a$ may be satisfying for a sample path $\omega$ and not for some other $\omega^{\prime}$; once the sample path is fixed, satisfying actions are unequivocally identified.

Example 3.5

Consider the multi-secretary problem with $T=5$, initial budget $B=2$, types $\Theta=\{1,2,3\}$ with $r_{1}>r_{2}>r_{3}$, and a particular sequence of arrivals $(\theta^{5},\theta^{4},\theta^{3},\theta^{2},\theta^{1})=(2,3,1,2,3)$. The optimal value of Offline is $r_{1}+r_{2}$, and this is achieved by accepting the sole type-$1$ arrival as well as any one out of the two type-$2$ arrivals. At time $t=5$, Offline is satisfied either accepting or rejecting $\theta^{5}$. Further, at $t=3$, for any budget $b>0$ the only satisfying action is to accept.

With the notion of satisfying actions, we can create a coupling as illustrated in Fig. 1. Although Offline may be satisfied with multiple actions (see above example and Remark 3.2), its value remains unchanged under any satisfying action, i.e., any tie-breaking rule. We define a valid policy $\pi^{\text{off}}$ for Offline to be any anticipatory functional such that, for every $\omega\in\Omega$, we have a different mapping to actions. Formally, for every $\omega\in\Omega$, $\pi^{\text{off}}[\omega]:[T]\times\mathcal{S}\times\Theta\to\mathcal{A}$ is a function satisfying the optimality principle:

[TABLE]

Next, we quantify by how much we need to compensate Offline when Online’s action is not satisfying, as follows

Definition 3.6** **(Marginal Compensation)

For action $a\in\mathcal{A}$, time $t\in[T]$ and state $s\in\mathcal{S}$, we denote the random variable $\partial R$ and scalar $\partial r$

[TABLE]

The random variable $\partial R$ captures exactly how much we need to compensate Offline, while $\partial r(a,j)$ provides a uniform (over $s,t$) bound on the compensation required when Online errs on an arrival of type $j$ by choosing an action $a$. Though there are several ways of bounding $\partial R(t,a,s)$, we choose $\partial r(a,j)$ as it is clean, expressive, and admits good bounds in many problems as the next example shows.

Example 3.7

For Online Packing problems define $r_{\max}\coloneqq\max_{j\in[n]}r_{j}$ as the maximum reward over all classes. The state space in this problem is $\mathcal{S}=\mathbb{N}^{d}$ and the actions space is $\mathcal{A}=\{\text{accept},\text{reject}\}$. Also, for simplicity, we assume that all resource requirements are binary, i.e., $a_{ij}\in\{0,1\}\,\forall\,i\in[d],j\in[n]$. For a given sample-path $\omega\in\Omega$ and any given budget $b\in\mathbb{N}^{d}$, if Offline decides to accept the arrival at $t$, we can instead make it reject the arrival while still earning a greater or equal reward by paying a compensation of $r_{\max}$. On the other hand, note that Offline can at most extract $r_{\max}$ in the future for every resource $\theta^{t}$ uses; hence on sample-paths where Offline wants to reject $\theta^{t}$, it can be made to accept $\theta^{t}$ instead with a compensation of $\lvert\lvert A_{\theta^{t}}\rvert\rvert_{1}r_{\max}\leq dr_{\max}$. In conclusion, we have $r_{j}\leq\partial r(a,j)\leq dr_{\max}$.

Recall that $S^{t}$ denotes the random process of Online’s state. Additionally, we denote $R^{\text{on}}(t,S^{t})[\omega]\coloneqq\mathcal{R}(\pi^{\text{on}}(t,S^{t},\theta^{t}),S^{t},\theta^{t})$ as the reward collected by Online at time $t$, and hence $V^{\textnormal{on}}(T,S^{T})[\omega]=\sum_{t\in[T]}R^{\text{on}}(t,S^{t})[\omega]$.

The final step is to fix Offline’s policy to be one which ‘follows Online’ as closely as possible. For this, given a policy $\pi^{\text{on}}$, on any sample-path $\omega$ we set $\pi^{\text{off}}(t,s,\theta^{t})[\omega]=\pi^{\text{on}}(t,s,\theta^{t})[\omega]$ if $\pi^{\text{on}}(t,s,\theta^{t})[\omega]$ is satisfying, and otherwise set $\pi^{\text{off}}(t,s,\theta^{t})[\omega]$ to an arbitrary satisfying action. In other words, we start with any valid policy $\pi^{\text{off}}$ and, for every $\omega\in\Omega$, we modify it as described to obtain another valid policy. Abusing notation, we still call $\pi^{\text{off}}$ this modified policy. Recall that this modification does not change the regret guarantees (see Remark 3.2).

Definition 3.8** **(Disagreement Set)

For any state $s$ and time $t$, and any action $a\in\mathcal{A}$, we define the disagreement set $Q(t,a,s)$ to be the set of sample-paths where $a$ is not satisfying for Offline, i.e.,

[TABLE]

Finally, let $Q(t,s)\subseteq\Omega$ be the event when Offline cannot follow Online, i.e., $Q(t,s)\coloneqq Q(t,\pi^{\text{on}}(t,s,\theta^{t}),s)$. Note that $Q(t,s)$ depends on $\pi^{\text{on}}$, but we omit the indexing since $\pi^{\text{on}}$ is clear from context. Only under $Q(t,s)$ we need to compensate Offline, hence we obtain the following.

Lemma 3.9** **(Compensated Coupling)

For any online decision-making problem, fix any Online policy $\pi^{\text{on}}$ with resulting state process $S^{t}$. Then we have:

[TABLE]

and thus $\mathbb{E}[\textsc{Reg}]\leq\max_{a\in\mathcal{A},j\in\Theta}\left\{\partial r(a,j)\right\}\cdot\sum_{t\in[T]}\mathbb{E}[\mathbb{P}[Q(t,S^{t})|S^{t}]]$.

Proof 3.10

We stress that, throughout, $S^{t}$ denotes Online’s state. We claim that, for every time $t$,

[TABLE]

To see this, let $a=\pi^{\text{on}}(t,S^{t},\theta^{t})$. If Offline is satisfied taking action $a$ in state $S^{t}$, then $V^{\textnormal{off}}(t,S^{t})[\omega]-V^{\textnormal{off}}(t-1,S^{t-1})[\omega]=R^{\text{on}}(t,S^{t})[\omega]$. On the other hand, if Offline is not satisfied taking action $a$, then by the definition of marginal compensation (Definition 3.6) we have, $V^{\textnormal{off}}(t,S^{t})[\omega]-V^{\textnormal{off}}(t-1,\mathcal{T}(a,S^{t},\theta^{t}))[\omega]=\partial R(t,a,S^{t})[\omega]+\mathcal{R}(a,S^{t},\theta^{t})$. Since by definition $\mathcal{T}(a,S^{t},\theta^{t})=S^{t-1}$ and $\mathcal{R}(a,S^{t},\theta^{t})=R^{\text{on}}(t,S^{t})$, we obtain Eq. 3. Finally, our first result follows by telescoping the summands and the second by linearity of expectation.

We list a series of remarks.

• •

Lemma 3.9 is a sample-path property that makes no reference to the arrival process. Though we use it primarily for analyzing MDPs, it can also be used for adversarial settings. We do not further explore this, but believe it is a promising avenue.

• •

For stochastic arrivals, the regret depends on $\mathbb{E}\left[\sum_{t\in[T]}\mathbb{P}[Q(t,S^{t})]\right]$; it follows that, if the disagreement probabilities are summable over all $t$, then the expected regret is constant. In Sections 4 and 5 we show how to bound $\mathbb{P}[Q(t,S^{t})]$ for different problems.

• •

The first part of Lemma 3.9 provides a distributional characterization of the regret in terms of a weighted sum of Bernoulli variables. This allows us to get high-probability bounds in Section 4.3.

• •

Lemma 3.9 gives a tractable way of bounding the regret which does not require either reasoning about the past decisions of Online, or the complicated process Offline may follow. In particular, it suffices to bound $\mathbb{P}[Q(t,S^{t})]$, i.e., the probability that, given state $S^{t}$ at time $t$, Offline loses optimality in trying to follow Online.

• •

As mentioned before, Lemma 3.9 extends to the full generality of MDPs. Indeed, from [26, chapter 11], it follows that any MDP with random transitions and random rewards can be simulated by the family of MDPs we study here; since the inputs $\theta^{t}$ are allowed to be random, we can define random transitions and rewards based on $\theta^{t}$, see [26] for further details.

Lemma 3.9 thus gives a generic tool for obtaining regret bounds against the offline optimum for any online policy. Note also that the compensated coupling argument generalizes to settings where the transition and reward functions are time dependent, the policies are random, etc. The compensated coupling also suggests a natural greedy policy, which we define next.

Comparison to traditional approaches.

As discussed in related work (Section 1.2), there are two main approaches. First, the fluid (or ex ante) benchmark, which can be understood as competing against a fixed value. This is the prevailing technique in competitive analysis [2] and the Online Packing literature [29, 36, 41]. We showed in Proposition 2.1 that such an approach cannot yield better than $O(\sqrt{T})$ regret bounds, while we prove $O(1)$. Second, the traditional sample path approach, which competes against the random trajectory of Offline (as illustrated in Fig. 1), is based on showing that Online is “close” to a fixed trajectory. This approach is capable of obtaining strong $O(1)$ guarantees [3], but it is highly involved since it necessitates a complete characterization of Offline’s trajectory. The benefit of our approach is abstracting away from the characterization of Offline’s trajectory and focusing only on Online’s (which is the one that the algorithm controls), while yielding strong $O(1)$ guarantees.

The next example illustrates the Compensated Coupling in a different setting. We consider the Ski Rental problem, which is a well studied minimum cost covering (not packing) problem.

Example 3.11** **(Ski Rental)

Given $T$ days for skiing, each day we decide whether to buy skis for $b$ dollars or to rent them for $1$ dollar. The snow may melt any day and we have a distribution over the period we may be able to ski, i.e., there is snow during the first $X\in[T]$ periods and we know the distribution of $X$. Our aim is to explicitly write the regret of a particular policy (stated later) using the Compensated Coupling.

The optimal offline solution is trivial: if $X<b$, Offline rents every day, otherwise ($X\geq b$) Offline buys the first day. In other words, Offline either buys the first day (which has cost $b$), or rents every day, with cost $X$. Since Offline knows $X$, he picks the minimum.

We map this problem to our framework as follows. The state space is $\mathcal{S}=\{\text{skis},\text{no-skis}\}$, where ‘skis’ means we own the skis. Arrivals are signals $\theta^{t}\in\{0,1\}$, where $1$ means there is snow and [math] that the season is over. The arrival sequence is always of the form $(\theta^{T},\ldots,\theta^{1})=(1,\ldots,1,0,\ldots,0)$, where $X=\sum_{t\in[T]}\theta^{t}$ by definition. Finally, rewards are $-1$ per day if we rent and $-b$ when we buy.

The compensations are as follows. If the state is ‘skis’ or if $\theta^{t}=0$ (season is over), then no compensation is needed, because we know that Offline does nothing w.p. 1 (either he owns the skis or the problem ended). The only case where Online and Offline may disagree is when $\theta^{t}=1$ (can ski today) and the state is ‘no-skis’.

Let us denote $X^{t}$ as the number of remaining skiing days (including $t$) and say we observe $\theta^{t}=1$ at time $t$. Offline is not satisfied renting if $X^{t}>b$; forcing him to rent in this event requires a compensation of $1$. On the other hand, Offline is not satisfied buying if $X^{t}<b$; forcing him needs a compensation of $b-X^{t}$. Consider the following policy $\pi^{\text{on}}$: for fixed $\tau\geq 0$, rent the first $\tau$ days and buy on day $\tau+1$ contingent on seeing snow all these days, i.e., contingent on $\theta^{t}=1$ for all $t=T,\ldots,T-\tau$. The Compensated Coupling (Lemma 3.9) allows us to write

[TABLE]

The first term corresponds to the disagreement of the first $\tau$ days (pay one dollar each day $t$ such that $X^{t}>b$), whereas the second is the disagreement of day $\tau+1$. This is an example where the compensated coupling yields an intuitive way of writing the regret. Furthermore, since the expression is exact, we can take expectations and optimize over $\tau$ to get the optimal policy (for example, see [4]).

3.3 The Bayes Selector Policy

Using the formalism defined in the previous sections, let $q(t,a,s)\coloneqq\mathbb{P}[Q(t,a,s)]$ be the disagreement probability of action $a$ at time $t$ in state $s$ (i.e., the probability that $a$ is not a satisfying action).

For any $t,s,\theta^{t}$, suppose we have an oracle that gives us $q(t,a,s)$ for every feasible action $a$. Given oracle access to $q(t,a,s)$ (or more generally, over-estimates $\hat{q}$ of $q$), a natural greedy policy suggested by Lemma 3.9 is that of choosing action $a$ that minimizes the probability of disagreement. This is similar in spirit to the Bayes selector (i.e., hard thresholding) in statistical learning. Algorithm 1 formalizes the use of this idea in online decision-making. The results below are essentially agnostic of how we obtain this oracle.

From Lemma 3.9, we immediately have the following:

Corollary 3.12** **(Regret Of Bayes Selector)

Consider Algorithm 1 with over-estimates $\hat{q}(t,a,s)\geq\mathbb{P}[Q(t,a,s)]\,\forall\,(t,a,s)$. If $A^{t}$ denotes the policy’s action at time $t$, then

[TABLE]

The next result states that, if we can bound the estimation error uniformly over states and actions, then the guarantee of the algorithm increases additively on the error (not multiplicatively, as one may suspect). In more detail, our next result is agnostic of the oracle used to obtain the the estimators $\hat{q}$. Examples of estimation procedures to obtain $\hat{q}$ include: simulation, function approximation, neural networks, etc. Regardless of how $\hat{q}$ is obtained, we can give a regret guarantee based only on the accuracy of the estimators. The following result follows as a special case of Corollary 3.12; we state it to emphasize that $\hat{q}$ can be estimated with some error.

Corollary 3.13** **(Bayes Selector with Imperfect Estimators)

Assume we have estimators $\hat{q}(t,a,s)$ of the probabilities $q(t,a,s)$ such that $\lvert q(t,a,s)-\hat{q}(t,a,s)\rvert\leq\Delta^{t}$ for all $t,a,s$. If we run Algorithm 1 with over-estimates $\hat{q}(t,a,s)+\Delta^{t}$, and $A^{t}$ denotes the policy’s action at time $t$, then

[TABLE]

Observe that, the total error induced due to estimation is a constant if, e.g., we can guarantee $\Delta^{t}=1/t^{2}$ or $\Delta^{t}=1/(T-t)^{2}$.

It is natural to consider a more sophisticated version of Algorithm 1, wherein we make decisions not only based on disagreement probabilities, but also take into account marginal compensations, i.e., the marginal loss of each decision. While Algorithm 1 is enough to obtain constant regret bounds in the problems we consider, we note that such an extension is possible and can be found in Section 10.2.

Remark 3.14

We discussed in Section 3 that the Compensated Coupling extends to the case where transitions and rewards can be random. By the same argument, the Bayes Selector (Algorithm 1) and its guarantees (Corollaries 3.12 and 3.13) extend too. Notice that Offline here is a prophet who has full knowledge of all the randomness (arrivals, transitions, and rewards).

4 Regret Guarantees for Online Packing

We now show that for the Online Packing problem, the Bayes Selector achieves an expected regret which is independent of the number of arrivals $T$ and the initial budgets $B$; in Section 5, we extend this to Online Matching problems.

In more detail, we prove that the dynamic fluid relaxation $(P_{t})$ in Eq. 4 provides a good estimator for the disagreement probabilities $q(t,a,s)$, and moreover, that the Bayes Selector based on these statistics reduces to a simple re-solve and threshold policy.

In this setting, the state space corresponds to resource availability, hence $\mathcal{S}=\mathbb{N}^{d}$; there are two possible actions, accept or reject, hence $\lvert\mathcal{A}\rvert=2$; finally, transitions correspond to the natural budget reductions given by the matrix $A$.

Recall that $Z(t)\in\mathbb{N}^{n}$ denotes the cumulative arrivals in the last $t$ periods and $B^{t}\in\mathbb{N}^{d}$ denotes Online’s budget at time $t$. Given knowledge of $Z(t)$ and state $B^{t}$, we define the ex-post relaxation $(P^{\star}_{t})$ and fluid relaxation $(P_{t})$ as follows.

[TABLE]

Remark 4.1

Offline* solves $(P^{\star}_{t})$ in Eq. 4, while Online solves $(P_{t})$. Both problems depend on Online’s budget at $t$; this is a crucial technical point and can only be accomplished due to the coupling we have developed.*

Let $X^{t}$ be a solution of $(P_{t})$ and $X^{\star t}$ a solution of $(P_{t}^{\star})$. Uniqueness of solutions is not required, see Proposition 4.8, and $X^{\star t}$ is for the analysis only. Our policy is detailed in Algorithm 2.

Intuitively, we ‘front-load’ (accept as early as possible) classes $j$ such that $X^{t}_{j}\geq\mathbb{E}[Z_{j}(t)]/2$ and back-load the rest (delay as much as possible). If Offline is satisfied accepting a front-loaded class (resp. rejecting a back-loaded class), he will do so. Accepting class $j$ is therefore an error if Offline, given the same budget as Online, picks no future arrivals of that class (i.e., $X^{\star t}_{j}<1$). On the other hand, rejecting $j$ is an error if $X^{\star t}_{j}>Z_{j}(t)-1$. We summarize this as follows:

Incorrect rejection: if $X^{t}_{j}<\frac{\mathbb{E}[Z_{j}(t)]}{2}$ and $X^{\star t}_{j}>Z_{j}(t)-1$. 2. 2.

Incorrect acceptance: if $X^{t}_{j}\geq\frac{\mathbb{E}[Z_{j}(t)]}{2}$ and $X^{\star t}_{j}<1$.

Observe that a compensation is paid only when the fluid solution is far off from the correct stochastic solution. In the proofs of Theorems 4.2 and 4.6 we formalize the fact that, since $X^{t}$ estimates $X^{\star t}$, such an event is highly unlikely – this along with the compensated coupling provides our desired regret guarantees.

Disagreement probabilities and the Fluid Bayes Selector.

The Bayes Selector (Algorithm 1) runs with over-estimates $\hat{q}(t,a,b)$ and, at each time, picks the minimum. On the other hand, Algorithm 2 is presented as the “simplified version”, in the sense that the decision rule is the one that minimizes suitable over-estimates $\hat{q}$. More importantly, we prove the following properties

[TABLE]

In other words, the property in Eq. 7 shows that Algorithm 2 is a Bayes Selector, while the property in Eq. 8 yields the desired constant regret bound in virtue of the Compensated Coupling and Corollary 3.12. We give explicit expressions for the values $\hat{q}$ and constants $c_{1},c_{2}$, see e.g. Eq. 21.

The robustness of the Bayes Selector.

The probability minimizing disagreement can be uniformly bounded over all budgets $b\in\mathbb{N}^{d}$, i.e., the exponential bound in Eq. 8 does not depend on $b$. This property has the following consequence: since the Fluid Bayes Selector has strong performance, many other Bayes Selector algorithms (using different $\hat{q}$) do too. In other words, the design of algorithms based on the Bayes Selector is robust and does not depend on “fine tunning” of the parameters $\hat{q}$. We make this precise in Corollaries 4.3 and 4.7 and uncover the same phenomenon for matching problems, see Corollary 5.2.

We need some additional notation before presenting our results. Let $\mathbb{E}_{j}[\cdot]$ ($\mathbb{P}_{j}[\cdot]$) be the expectation (probability) conditioned on the arrival at time $t$ being of type $j$, i.e.,  $\mathbb{P}_{j}[\cdot]=\mathbb{P}[\cdot|\theta^{t}=j]$. We denote $r_{\max}\coloneqq\max_{j\in[n]}r_{j}$ and $p_{\min}\coloneqq\min_{j\in[n]}p_{j}$.

4.1 Special Case: Multi-Secretary with Multinomial Arrivals

Before we proceed to the general case, we state the result for the Multi-Secretary problem. We present this result separately because in this one dimensional problem we can obtain a better and explicit constant. The proofs of Theorem 4.2 and Corollary 4.3 below can be found in Appendix 10.3.

Theorem 4.2

The expected regret of the Fluid Bayes Selector (Algorithm 2) for the multi-secretary problem with multinomial arrivals is at most $r_{\max}\sum_{j>1}2/p_{j}\leq 2(n-1)r_{\max}/p_{\min}$.

This recovers the best-known expected regret bound for this problem shown in a recent work [3]. However, while the result in [3] depends on a complex martingale argument, our proof is much more succinct, and provides explicit and stronger guarantees; in particular, in Section 4.3, we provide concentration bounds for the regret.

Moreover, Theorem 4.2, along with Corollary 3.13, provides a critical intermediate step for characterizing the performance of Algorithm 1 for the multi-secretary problem.

Corollary 4.3

For the multi-secretary problem with multinomial arrivals, the expected regret of the Bayes Selector (Algorithm 1) with any imperfect estimators $\hat{q}$ is at most $2r_{\max}\left(\sum_{j>1}1/p_{j}+\sum_{t\in[T]}\Delta^{t}\right)$, where $\Delta^{t}$ is the accuracy defined by $\lvert q(t,a,b)-\hat{q}(t,a,b)\rvert\leq\Delta^{t}$ for all $t\in[T],a\in\mathcal{A},b\in\mathbb{N}$.

Observe that, if $\Delta^{t}$ is summable, e.g., $\Delta^{t}=1/t^{2}$ or $\Delta^{t}=1/(T-t)^{2}$, then Corollary 4.3 implies constant expected regret for all these types of estimators we can use in Algorithm 1.

4.2 Online Packing with General Arrivals

We consider now the case $d>1$ and arrival processes other than Multinomial. We assume the following condition on the process $Z(t)$, which we refer to as All-Time Deviation.

Definition 4.4** **(All-Time Deviation)

Let $\mu$ be a given norm in $\mathbb{R}^{n}$ and $\kappa\in\mathbb{R}_{\geq 0}^{n}$ a constant parameter. An $n$ dimensional process $Z(t)$ satisfies the All-Time Deviation bound w.r.t. $\mu$ and $\kappa$ if, for all $j\in[n]$, there are constants $c_{j}\geq 0$ and naturals $\tau_{j}$ such that

[TABLE]

We remark that we do not need exponential tails, as it is common to assume, but rather a simple quadratic tail. Additionally, some common tail bounds are valid only for large enough samples; the parameters $\tau_{j}$ capture this technical aspect. In this section we will use the definition with $\kappa_{j}$ the same entry for all $j$, thus denoted simply by $\kappa>0$. In Section 5 we require the definition with the more general form.

Example 4.5** **(Multinomial and Poisson tails)

In these examples we actually have the stronger exponential tails, so we do not elaborate on the constant $c_{j}$.

For multinomial arrivals, [21, Lemma 3] guarantees

[TABLE]

By setting $\varepsilon=p_{j}/2\kappa$, we conclude that Definition 4.4 is satisfied with constants $\tau_{j}=(p_{j}/2\kappa)^{2}n/20$.

For Poisson arrivals, from the proof of [21, Lemma 3], $\mathbb{P}(\lvert X-\lambda\rvert\geq\varepsilon\lambda)\leq 2e^{-\lambda\varepsilon^{2}/4}$ is valid for $X\sim\operatorname{Poisson}(\lambda)$ and any $\varepsilon>0$. Using this, we can simply take $\tau_{j}=0$.

In the remainder of this subsection we generalize our ideas to prove the following.

Theorem 4.6

Assume the arrival process $(Z(t):t\in[T])$ satisfies the conditions in Eq. 9. The expected regret of the Fluid Bayes Selector (Algorithm 2) for Online Packing is at most $dr_{\max}M$, where $M$ is independent of $T$ and $B$. Specifically, for $\kappa=\kappa(A)$ we have

For Multinomial arrivals: $M\leq 103\kappa^{2}\sum_{j\in[n]}1/p_{j}$. 2. 2.

For general distributions satisfying Eq. 9: $M\leq\sum_{j\in[n]}p_{j}(2c_{j}+\max\{\tau_{j},\tilde{\tau}_{j}\})$, where $p_{j}$ is an upper bound on $\mathbb{P}[\theta^{t}=j]$ and $\tilde{\tau}_{j}$ is such that $\mathbb{E}[Z_{j}(\tilde{\tau}_{j})]\geq 2$, i.e., it is large enough.

The constant $\kappa(A)$ is given by Proposition 4.8 below. Just as before, Theorem 4.6, along with Corollary 3.13, provides a performance guarantee for Algorithm 1. We state the corollary without proof, since it is identical to that of Corollary 4.3.

Corollary 4.7

For the Online Packing problem, if the arrival process satisfies the conditions in Eq. 9, the expected regret of the Bayes Selector (Algorithm 1) with any imperfect estimators $\hat{q}$ is at most $dr_{\max}(M+2\sum_{t\in[T]}\Delta^{t})$, where $M$ is as in Theorem 4.6 and $\Delta^{t}$ is the accuracy defined by $\lvert q(t,a,b)-\hat{q}(t,a,b)\rvert\leq\Delta^{t}$ for all $t\in[T],a\in\mathcal{A},b\in\mathbb{N}^{d}$.

To prove Theorem 4.6, we need to quantify how the change in the right-hand side of an LP impacts optimal solutions. Indeed, as stated in Eq. 4, the solutions $X^{t}$ and $X^{\star t}$ correspond to perturbed right-hand sides ($\mathbb{E}[Z(t)]$ and $Z(t)$ respectively). The following proposition implies that small changes in the arrivals vector do not change the solution by much and it is based on a more general result from [32, Theorem 2.4].

Proposition 4.8** **(LP Lipschitz Property)

Given $b\in\mathbb{R}^{d}$, and any norm $\lvert\lvert\cdot\rvert\rvert_{\mu}$ in $\mathbb{R}^{n}$, consider the following LP

[TABLE]

Then $\exists$ constant $\kappa=\kappa_{\mu}(A)$ such that, for any $y,\hat{y}\in\mathbb{R}_{\geq 0}^{n}$ and any solution $x$ to $P(y)$, there exists a solution $\hat{x}$ solving $P(\hat{y})$ such that $\lvert\lvert x-\hat{x}\rvert\rvert_{\infty}\leq\kappa\lvert\lvert y-\hat{y}\rvert\rvert_{\mu}$.

Proof 4.9** **(Proof of Theorem 4.6)

Recall the two conditions derived from our decision rule: (1) Incorrect rejection of $j$ means $X^{t}_{j}<\mathbb{E}[Z_{j}(t)]/2$ and $X^{\star t}_{j}>Z_{j}(t)-1$. (2) Incorrect acceptance of $j$ means $X^{t}_{j}\geq\mathbb{E}[Z_{j}(t)]/2$ and $X^{\star t}_{j}<1$. We have to additionally account for feasibility, i.e., we can only accept a request $j$ if $B_{i}^{t}\geq a_{ij}$ for all $i\in[d]$. In case there are not enough resources, our decision rule is feasible if either $X^{t}_{j}<\mathbb{E}[Z_{j}(t)]/2$ (reject) or $X^{t}_{j}\geq 1$ (since $X^{t}$ is feasible for $(P_{t})$). Only in the case $X^{t}_{j}\geq\mathbb{E}[Z_{j}(t)]/2$ and $X^{t}_{j}<1$ we need to disregard our decision rule and are forced to reject; under such a condition we must pay a compensation of $r_{j}$. Observe that this condition is never met if $\mathbb{E}[Z_{j}(t)]\geq 2$, i.e., it is vacuous for $t\geq\tilde{\tau}_{j}$.

The disagreement sets (Definition 3.8) are thus $Q(t,b)=\{\omega\in\Omega:\text{either }(1),(2)\text{ or }t<\tilde{\tau}_{j}\}$, where (1) and (2) are the previous conditions. Now we can upper bound the probability of paying a compensation as follows. Call $E_{j}$ the event $\{\omega\in\Omega:\lvert\lvert Z(t)-\mathbb{E}(Z(t))\rvert\rvert_{1}\leq\mathbb{E}[Z_{j}(t)]/2\kappa\}$. In this event, Proposition 4.8 implies $\lvert X^{t}_{j}-X^{\star t}_{j}\rvert\leq\mathbb{E}[Z_{j}(t)]/2$, hence conditions (1) and (2) do not happen when $E_{j}$ occurs, i.e., $\mathbb{P}_{j}[Q(t,b)|E_{j}]\leq\mathds{1}_{\left\{t<\tilde{\tau}_{j}\right\}}$. Observe that $\mathbb{P}[\bar{E}_{j}]\leq f_{j}(t)+\mathds{1}_{\left\{t<\tau_{j}\right\}}$, where $f_{j}(t)=c_{j}/t^{2}$ for general processes satisfying Eq. 9 and $f_{j}(t)=e^{-t(p_{j}/2\kappa)^{2}/25}$ for the Multinomial process (see Eq. 10). Finally,

[TABLE]

Summing up over time, we get

[TABLE]

Since $\sum_{t\in[T]}1/t^{2}\leq\pi^{2}/6\leq 2$, this finishes the proof for general processes. For the case of Multinomial arrivals, we can be more refined. Indeed, $\tilde{\tau}_{j}$ is defined by $\mathbb{E}[Z_{j}(\tilde{\tau}_{j})]\geq 2$, i.e., $\tilde{\tau}_{j}\geq 2/p_{j}$ and $\tau_{j}=(p_{j}/2\kappa)^{2}n/20$ (see Eq. 10). From the previous equation, with the stronger exponential bound $f_{j}(t)=e^{-t(p_{j}/2\kappa)^{2}/25}$ we get

[TABLE]

Since $n\leq\sum_{j\in[n]}\frac{1}{p_{j}}$, we arrive a the desired bound. The result follows via the compensated coupling (Lemma 3.9) and Corollary 3.12.

Remark 4.10

In the multi-secretary problem it is easy to conclude $\kappa(A)=1$, thus this analysis recovers the same bound up to absolute constants (namely $103$ vs $2$). The larger constant comes exclusively from the larger constants in the tail bounds of Multinomial compared to Binomial r.v.

Remark 4.11

More refined bounds on $M$ can be obtained by not bounding $\mathbb{P}[\theta^{t}=j]\leq p_{j}$, but rather by $\mathbb{P}[\theta^{t}=j]\leq p_{j}(t)$. For example, a time-varying version of a Multinomial process easily fits in our framework and the proof does not change.

Remark 4.12

The theorem holds even under Markovian correlations (see Example 4.13 below), where the distribution of $Z(t-1)$ depends on $\theta^{t}$. It is interesting that in this case it is impossible to run the optimal policy for even moderate instance sizes, since the state space is huge, while the Bayes Selector still offers bounded expected regret.

We now give two examples of other arrival processes that satisfy the All-Time Deviation (Definition 4.4). The proofs of the bounds are short, but we relegate them to Appendix 10.4. We emphasize that Example 4.14 below has quadratic tails (instead of exponential), hence we term it heavy tailed.

Example 4.13** **(Markovian Arrival Processes)

We consider the case where $\theta^{t}$ is drawn from an ergodic Markov chain. Let $P\in\mathbb{R}_{\geq 0}^{n\times n}$ be the corresponding matrix of transition probabilities. The process unfolds as follows: at time $t=T$ an arrival $\theta^{T}\in[n]$ is drawn according to an arbitrary distribution, then for $t=T,\ldots,2$ we have $\mathbb{P}[\theta^{t-1}=j|\theta^{t}]=P_{\theta^{t}j}$. Let $\nu\in\mathbb{R}_{\geq 0}^{n}$ be the stationary distribution. We do not require long-run or other usual stationary assumptions; the process is still over a finite horizon $T$. This process satisfies All-Time Deviation with exponential tails. Specifically, with the norm $\mu=\lvert\lvert\cdot\rvert\rvert_{\infty}$, for some constants $c_{j},c^{\prime}$ that depend on $P$ only we have

[TABLE]

Example 4.14** **(Heavy Tailed Poisson Arrivals)

We consider the case where the arrival process is governed by independent time varying Poisson processes with arrival rates $\lambda_{j}(t)>0$, which we assume for simplicity have finitely many discontinuity points (so that all the expectations are well defined). Under the following conditions, the process satisfies the All-Time Deviation with quadratic tails and norm $\mu=\lvert\lvert\cdot\rvert\rvert_{\infty}$.

[TABLE]

In other words, we require $f(t)=\omega(1)$ and $g(t)$ is any function. Intuitively, Eq. 13 guarantees that no type $j$ “overwhelms” all other types; observe that, when the rates are constant, this is trivially satisfied with $g(t)$ constant. On the other hand, Eq. 14 controls the minimum arrival rate, which can be as small as $\omega(\log(t)/t)$. Observe that our conditions allow for the intensity to increase closer to the end ($t=0$), i.e., we incorporate the case where agents are more likely to arrive closer to the deadline.

4.3 High-Probability Regret Bounds

We have proved that $\mathbb{E}[\textsc{Reg}]$ is constant for packing problems. One may worry that this is not enough because, since it is a random variable, Reg may still realize to a large value. We present a bound for the distribution of Reg showing that it has light tails.

Proposition 4.15

For packing problems, there are constants $\tau$ and $c_{j}$ for $j\in[n]$, depending on $A,p$ and the distribution of $Z$ only, such that

For Multinomial or Poisson arrivals: $\forall x>\tau$, $\mathbb{P}[\textsc{Reg}>x]\leq\sum_{j}p_{j}e^{-c_{j}x/r_{\max}}/c_{j}$. 2. 2.

For general distributions satisfying Eq. 9: $\forall x>\tau$, $\mathbb{P}[\textsc{Reg}>x]\leq\frac{r_{\max}}{x}\sum_{j}p_{j}c_{j}$.

The proof is based on the following simple lemma. The idea is to first bound the disagreements of our algorithm, as defined in Section 3.3. The total number of disagreements is a sum of dependent Bernoulli variables, which we bound next.

Lemma 4.16

Let $\{X^{t}:t\in[T]\}$ be a sequence of dependent r.v. such that $X^{t}\sim\operatorname{Bernoulli}(p_{t})$ and let $\{q_{t}:t\in[T]\}$ be numbers such that $q_{t}\geq p_{t}$. If we define $D\coloneqq\sum_{t=1}^{T}X^{t}$, then

[TABLE]

Proof 4.17

Fix $d\in[T]$ and observe that

[TABLE]

Indeed, if the condition $(\exists t\geq d,X^{t}=1)$ fails, then at most $d-1$ variables $X^{t}$ can be one.

Finally, a union bound shows $\mathbb{P}[D\geq d]\leq\sum_{t\geq d}\mathbb{P}[X^{t}=1]$. Since $q_{t}\geq p_{t}$, the proof is complete.

Proof 4.18** **(Proof of Proposition 4.15)

As described in the previous subsections, we can write $\textsc{Reg}\leq r_{\max}D$, with $D$ the number of disagreements. Additionally, $D$ is a sum of $T$ Bernoulli r.v. $X^{t}$, each with parameter bounded by $q_{t}$.

In the case of Multinomial and Poisson r.v., as described in Section 4.2, we have exponential bounds $q_{t}\leq\sum_{j\in[n]}p_{j}e^{-c_{j}t}$ for $t\geq\tau=\max_{j\in[n]}\tau_{j}$. We conclude invoking Lemma 4.16 and upper bounding $\sum_{t=x+1}^{T}e^{-c_{j}t}\leq e^{-c_{j}x}/c_{j}$.

For general distributions, as described in Section 4, we have the bounds $q_{t}\leq\sum_{j\in[n]}p_{j}\frac{c_{j}}{t^{2}}$ for $t\geq\tau$. Using Lemma 4.16 and bounding $\sum_{t=x+1}^{T}t^{-2}\leq 1/x$ finishes the proof.

5 Regret Guarantees for Online Matching

We turn to an alternate setting, where each incoming arrival corresponds to a unit-demand buyer – in other words, each arrival wants a unit of a single resource, but has different valuations for different resources. This is essentially equivalent to the online bipartite matching problem with edge weights (weights correspond to rewards) where there can be multiple copies of each node.

As before, we are given a matrix $A\in\{0,1\}^{d\times n}$ characterizing the demand for resources, which can be interpreted as the adjacency matrix in the Online Matching problem. Define $S_{j}\coloneqq\{i\in[d]:a_{ij}=1\}$. If we allocate any resource $i\in S_{j}$ to an agent type $j$, we obtain a reward of $r_{ij}$, whereas allocating $i\not\in S_{j}$ has no reward. We can allocate at most one item to each agent.

Given resource availability $B\in\mathbb{N}^{d}$ and total arrivals $Z\in\mathbb{N}^{n}$, we can formulate Offline’s problem as follows, where the variable $x_{ij}$ denotes the number of items $i$ allocated to agents of type $j$.

[TABLE]

We assume that the process $Z(t)$ satisfies the All-Time Deviation bound (see Definition 4.4) w.r.t. the one-norm and parameters $\kappa_{j}=(\lvert S_{j}\rvert+1)/2$. This condition can be restated as follows. For every $j\in[n]$, there are constants $c_{j}\geq 0$ and naturals $\tau_{j}$ such that

[TABLE]

We now state the main result of this section, which is based on an instantiation of the Bayes Selector. As before, the theorem readily implies performance guarantees for Algorithm 1, which we state without proof, since it is identical to that of Corollary 4.3.

Theorem 5.1

For the Online Matching problem, if the arrival process satisfies the conditions in Eq. 16, then the expected regret of the Fluid Bayes Selector (Algorithm 3) is at most $r_{\max}\sum_{j\in[n]}p_{j}(c_{j}+\tau_{j})$, where $p_{j}$ is an upper bound on $\mathbb{P}[\theta^{t}=j]$.

Corollary 5.2

For the Online Matching problem, if the arrival process satisfies the conditions in Eq. 16, then the expected regret of the Bayes Selector (Algorithm 1) with any imperfect estimators $\hat{q}$ is at most $r_{\max}(M+2\sum_{t\in[T]}\Delta^{t})$. The constant $M=\sum_{j\in[n]}p_{j}(c_{j}+\tau_{j})$ is as in Theorem 5.1 and $\Delta^{t}$ is the accuracy defined by $\lvert q(t,a,s)-\hat{q}(t,a,s)\rvert\leq\Delta^{t}$.

5.1 Algorithm and Analysis

We start from the LP in Eq. 15, then add a fictitious item $d+1$ which no agent wants with initial budget $B^{T}_{d+1}=T$; now all agents are matched, but, if we match an agent to $d+1$, there is no reward. Using the Compensated Coupling, we can write two coupled optimization problems, $(P_{t}^{\star})$ for Offline and $(P_{t})$ for Online as follows.

[TABLE]

Recall that $B^{t}$ represents Online’s budget with $t$ periods to go. We solve $(P_{t})$ in Eq. 17 and obtain an optimizer $X^{t}$. If $\theta^{t}=j$, let $K\in\operatorname{argmax}\{X^{t}_{i,j}:i\in[d+1]\}$ be the maximal entry, breaking ties arbitrarily, then match $j$ to $K$. The resulting policy is presented in Algorithm 3. Observe that, matching an agent to $K=d+1$ (fictitious resource) is equivalent to rejecting him.

Disagreement Sets.

At each time $t$, matching $\theta^{t}=j$ to $K$ requires a compensation only if Offline never matches a type $j$ to $K$, i.e., $X^{\star t}_{K,j}<1$. On the other hand, Algorithm 3 picks $K$ to be the largest component, hence we should have $X^{t}_{K,j}>>1$ (precisely, $X^{t}_{K,j}\geq\frac{\mathbb{E}[Z_{j}(t)]}{d+1}$). More formally, the constraint $\sum_{i\in[d+1]}x_{ij}=\mathbb{E}[Z_{j}(t)]$ in Eq. 17 and the definition of $S_{j}$ imply $X^{t}_{K,j}\geq\mathbb{E}[Z_{j}(t)]/(\lvert S_{j}\rvert+1)$. We conclude that, if matching to $K$ is not satisfying (see Definition 3.4), it must be that $\lvert\lvert X^{t}-X^{\star t}\rvert\rvert_{\infty}>\mathbb{E}[Z_{j}(t)]/(\lvert S_{j}\rvert+1)$. Proposition 5.3 below characterizes exactly this deviation.

Observe that, in Eqs. 15 and 17, the matrix $A$ appears only on the objective function; this is not the usual LP formulation for this problem, but it allows us to obtain the following result. We remark that not only we have a Lipschitz property, but the Lipschitz constant is exactly $1$. We present the proof of Proposition 5.3 in Section 10.5.

Proposition 5.3** **(Lipschitz Property for Matching)

Take any $z^{1},z^{2}\in\mathbb{R}_{\geq 0}^{d}$ and $b\in\mathbb{R}_{\geq 0}^{d}$. If $\mathbf{x}^{1}$ is a solution of $P[z^{1},b]$, then there exists $\mathbf{x}^{2}$ solving $P[z^{2},b]$ such that $\lvert\lvert\mathbf{x}^{1}-\mathbf{x}^{2}\rvert\rvert_{\infty}\leq\lvert\lvert z^{1}-z^{2}\rvert\rvert_{1}$.

From here, the proof of Theorem 5.1 is applying the Compensated Coupling (Lemma 3.9) and Corollary 3.12 in the same way as we did in Section 4, hence we omit it.

5.2 Online Stochastic Matching

A classical problem that fits naturally into the above framework is that of online bipartite matching problem with stochastic inputs [33]. The reader unfamiliar with the problem can find the details of the setup in Section 10.6. For this setting, the bound obtained via compensated coupling surprisingly holds with equality:

Lemma 5.4

For the stochastic online bipartite matching, given an online policy, if $U^{t}$ denotes the node matched at time $t$ by Online and $S^{t}$ the available nodes, then

[TABLE]

Based on this, it is tempting to conjecture that the Bayes Selector does in fact lead to an optimal policy for this setting. This however is not the case, although showing this is surprisingly subtle; in Appendix 10.6, we discuss this in more detail. Moreover, it is known that this problem cannot admit an expected regret that has better than linear scaling with $T$ (in particular, [33] proves a constant upper bound on the competitive ratio for this setting). That said, the strength of the above bound suggests that the Bayes selector may have strong approximation guarantees – showing this remains an open problem.

6 Regret Guarantees for Online Allocation

We now give the algorithm and analysis for the general Online Allocation problem defined in Section 2.1. As before, let us introduce a fictitious resource $i=d+1$ with initial capacity $B_{d+1}=T$, zero rewards ($r_{\{d+1\}j}=0$ for all $j\in[n]$) and such that $\{d+1\}\in S_{j}$ for all $j\in[n]$. Now we can assume w.l.o.g. that each agent gets assigned a bundle. Finally, for a bundle $s$, we denote $a_{is}\in\mathbb{N}$ the number of times the resource $i$ appears in $s$ (recall that bundles are multisets).

Given resource availability $B\in\mathbb{N}^{d+1}$ and total arrivals $Z\in\mathbb{N}^{n}$, we can formulate the coupled problems for Offline and Online as follows, where the variable $x_{sj}$ denotes the number of times a bundle $s\in S_{j}$ is allocated to a type $j$.

[TABLE]

We assume that the process $Z(t)$ satisfies the All-Time Deviation bound (see Definition 4.4) w.r.t. some norm $\mu$ and parameters $\kappa_{j}=(d+1)\kappa$, where $\kappa=\kappa_{\mu}(A)$ depends only on $A$ and $\mu$. This condition can be restated as follows. For every $j\in[n]$, there are constants $c_{j}\geq 0$ and naturals $\tau_{j}$ such that

[TABLE]

We present the resulting policy in Algorithm 4 with its guarantee in Theorem 6.1. We remark that the constant $\kappa$ depends only on the constraint matrix defining the LP in Eq. 18, i.e., it does depend on the choices of bundles $S_{j}$, but it is independent of $T$ and $B$.

Theorem 6.1

For the Online Allocation problem, there exists a constant $\kappa$ that depends on $(S_{j}:j\in[n])$ only such that, if the arrival process satisfies the conditions in Eq. 19, then the expected regret of the Fluid Bayes Selector (Algorithm 4) is at most $r_{\max}\sum_{j\in[n]}p_{j}(c_{j}+\tau_{j})$, where $p_{j}$ is an upper bound on $\mathbb{P}[\theta^{t}=j]$.

The proof of Theorem 6.1 is analogous to that of Theorem 4.6, hence we omit it and provide here only the key steps. Recall that, for request $j$, since we include the fictitious item, there are are $\lvert S_{j}\rvert+1$ possible bundles. Crucially, incorrect allocation of $s$ to $j$ necessitates $X^{t}_{sj}\geq\mathbb{E}[Z_{j}(t)]/(\lvert S_{j}\rvert+1)$ (because Algorithm 4 takes the maximum entry) and $X^{\star t}_{j}<1$ (Offline never allocates $s$ to $j$). By the Lipschitz property of LPs (see Proposition 4.8), this event requires a large deviation of $Z(t)$ w.r.t. its mean, which has low probability. More formally, the disagreement sets (Definition 3.8) are $Q(t,b)=\{\omega\in\Omega:X^{t}_{sj}\geq\mathbb{E}[Z_{j}(t)]/(\lvert S_{j}\rvert+1)\text{ and }X^{\star t}_{j}<1\}$. By the Lipschitz property, $Q(t,b)\subseteq\{\omega\in\Omega:\lvert\lvert Z(t)-\mathbb{E}(Z(t))\rvert\rvert_{\mu}\geq\frac{\mathbb{E}[Z_{j}(t)]}{\kappa(\lvert S_{j}\rvert+1)}\}$. The probability of this last event is bounded by Eq. 19, hence the Compensated Coupling concludes the proof.

7 Numerical Experiments

The theoretical results we have presented, together with known lower bounds for previous algorithms, show that our approach outperforms existing heuristics for Online Packing and Online Matching problems. We now re-emphasize these results via simulation with synthetic data, which demonstrates both the sub-optimality of existing heuristics (in terms of expected regret which scales with $T$), as well as the fact that the Bayes selector has constant expected regret.

We run experiments for both Online Packing and Online Matching with multinomial arrivals. For each problem we consider two instances, i.e., two sets of parameters $(r,A,p)$, then we scale each instance to obtain a family of ever larger systems. For each scaling we run 100 simulations. In conclusion, we run four sets of parameters (two for packing and two for matching), each scaled to generate many systems. The code for all the algorithms can be found in https://github.com/albvera/bayes_selector.

7.1 Online Packing

We compare the Bayes Selector against three policies: (i) Static Randomized (SR) is the first known policy with regret guarantees, it is based on solving the fluid LP once and using the solution as a randomized acceptance rule [37]. (ii) Re-solve and Randomize is based on re-solving the fluid LP at each time and using the solution as a randomized acceptance rule [29]. (iii) Infrequent Re-solve with Thresholding (IRT) is based on re-solving the fluid LP at carefully chosen times, specifically at times $\{T^{(5/6)^{u}}:u=0,1,...,\log\log(T)/\log(6/5)\}$, then either randomize or threshold depending on the value of the solution [13].

Our first instance has $d=2$ resources and $n=6$ agent types. Types $j\in\{1,2\}$ require one unit of resource $i=1$, types $j\in\{3,4\}$ require one unit of $i=2$, and types $j\in\{5,6\}$ require one unit of each resource. All the parameters are presented in Table 1. We consider a base system with capacities $B_{1}=B_{2}=40$ and horizon $T=200$. The base system is chosen such that the problem is near dual-degenerate (which is the regime where heuristics based on the fluid benchmark are known to have poor performance; see Proposition 2.1). Finally, for a scaling $k\in\mathbb{N}$, the $k$-th system has capacities $kB$ and horizon $(k+k^{0.7})T$. We remark that traditionally the horizon is scaled as $kT$, but we chose this slightly different scaling to emphasize that our result does not depend on the specific way the system is scaled.

The results for the first instance are summarized in Fig. 2, where we also present a log-log plot which allows better to appreciate how the regret grows. Static Randomized has the worst performance in our study; indeed, we do not include it in the plot since it is orders of magnitude higher. We note that not only the Bayes Selector outperforms previous methods, but the regret is very small (both in average and sample-path wise), specially in comparison with the overall reward which grows linearly with $k$, i.e. $V^{\textnormal{off}}=\Omega(k)$ (in expectation and w.h.p.).

The second instance has $n=15$ agent types and $d=20$ resources, the specific parameters are presented in Table 3 in Section 11 and were generated randomly. We take a base system with horizon $T=50$ and capacities $B_{i}=10$ for all $i\in[20]$, then the $k$-th system has horizon $kT$ and capacities $kB$. The performance of different algorithms is presented in Fig. 3. We notice that this instance is not degenerate and we are scaling linearly, hence all the algorithms except Static Randomize (which we again omit from the plots) are known to achieve constant regret. Nevertheless, we observe that the Bayes Selector has the best performance by a large margin.

7.2 Online Matching

As we mentioned in Section 5, our problem corresponds to stochastic matching with edge weights. There has been previous work studying constant factor approximations for worst-case distributions. In particular, the state of the art is a $0.705$ competitive ratio [10], while a previous algorithm achieved a $0.667$ competitive ratio [25]. Both algorithms are impractical since they require a sampling procedure over $\text{poly}(T\cdot\max_{i\in[d]}B_{i})$ many matchings. To the best of our knowledge, the best guarantee of a practical algorithm is a $1-1/e\approx 0.63$ competitive ratio and is achieved by the base algorithm in [25] (that when built upon achieves the $0.667$ guarantee). We therefore benchmark against this algorithm, which we call “Competitive”.

Competitive is based on solving a big LP once (it has $\Omega(T\cdot\max_{i\in[d]}B_{i})$ variables) and using the solution as a probabilistic allocation rule. We also compare against a contemporaneous algorithm, called Marginal Allocation, that is based on bid-prices [40]. Marginal Allocation uses approximate dynamic programming to obtain the marginal benefit of a matching, then uses this marginal value as a bid price so that, if the reward exceeds it, then we match the request. We give further details for both Marginal Allocation and Competitive in Section 11.

The first instance we consider has $d=2$ resources and $n=6$ agent types. The specific parameters are presented in Table 2, where reward $r_{ij}=0$ implies that type $j$ cannot be matched to that resource $i$, i.e., $A_{ij}=0$. We consider a base system with horizon $T=20$ and capacities $B=(4,5)^{\prime}$ and then scale it linearly so that the $k$-th system has horizon $kT$ and capacities $kB$. Our second instance has $d=6$ resources and $n=10$ agent types, the specific parameters are presented in Table 4 in Section 11. We consider a base system with horizon $T=200$ and capacities $B=(40,50,40,30,20,40)^{\prime}$ and then scale it linearly so that the $k$-th system has horizon $kT$ and capacities $kB$.

The results are presented in Fig. 4. We do not include the regret of Competitive, since it is so high that it distorts the plots (it starts at 80 times the regret of the other algorithms and then grows linearly with $k$). We can confirm that the Bayes Selector has constant regret and, additionally, offers the best performance. Marginal Allocation offers a much better performance than Competitive, but its regret still grows and seems to scale as $\Omega(\sqrt{T})$.

8 Conclusions

We reiterate that our contributions in this paper are both to develop new online policies that achieve constant regret for a large class of online resource allocation problems, and also, a new technique for analyzing online decision-making heuristics.

Our work herein has developed a new technical tool—the Compensated Coupling—for analyzing online decision making policies with respect to offline benchmarks. In short, the main insight is that, through the use of compensations, we can couple Offline’s state to that of Online on every sample path. This simplifies the analysis of online policies since, in contrast to existing approaches, we do not need to track the complicated offline process.

Next, we presented a general class of problems, which we referred to as Online Allocation, wherein different agents request different bundles of resources. This problem captures, among others, Online Packing and Online Matching. For all of these problems we present a tractable policy, the Bayes Selector, based on re-solving an LP, that achieves constant regret.

Our analysis is based on the Compensated Coupling and, thanks to its versatility, we can accommodate a large class of arrival processes including: correlated processes, heavy tailed, and the classical Poisson and Multinomial (i.i.d.).

Although we instantiate the Bayes Selector for Online Allocation, we defined it for general MDPs; we hope this policy is useful for other types of problems too. We remark two properties of the Bayes Selector: (i) it works on interpretable quantities, namely the estimation of disagreement probabilities $\hat{q}$, and (ii) it is amenable to simulation, since $\hat{q}$ can be estimated by running offline trajectories. We therefore think that a promising avenue for further research is to apply this policy to other problems using modern estimation techniques.

The assumption of finite types of agents is well founded in revenue management problems, but there are settings where the number of types could be very large or even continuous. Based on reported numerical results [3], the Bayes Selector appears to have good performance even in this setting. An interesting problem is to obtain parametric guarantees (not worst case) in the case where the number of types is very large or continuous.

Acknowledgement

We gratefully acknowledge support from the ARL under grant W911NF-17-1-0094, and the NSF under grant DMS-1839346 and ECCS-1847393. A preliminary version of this work was presented at SIGMETRICS 2019 and accompanied by a one-page extended abstract [39]. We thank the anonymous reviewers for their helpful feedback.

9 The Fluid Benchmark

Proof 9.1** **(Proof of Proposition 2.1)

To build intuition, we start with a description of dual degeneracy for the online knapsack problem with budget $B\leq T$. We assume w.l.o.g. $r_{1}\geq r_{2}\geq\ldots\geq r_{n}$ and denote $Z=Z(T)$. The primal and dual are given by

[TABLE]

Let us denote $\mu\coloneqq\mathbb{E}[Z]$. If the fluid $(P[\mu])$ is degenerate, then we have $n+1$ active constraints. It is straightforward to conclude that there must be an index ${j^{\star}}$ such that $\sum_{j\leq{j^{\star}}}\mathbb{E}[Z_{j}]=B$. The fluid solution is thus $x_{j}=\mathbb{E}[Z_{j}]$ for $j\leq{j^{\star}}$ and $x_{j}=0$ for $j>{j^{\star}}$. We can construct two dual solutions as follows. Let $\alpha^{1}=r_{j^{\star}}$ and $\alpha^{2}=r_{{j^{\star}}+1}$, these correspond to the shadow prices for alternative budgets $B-\varepsilon$ and $B+\varepsilon$ respectively. The corresponding variables $\beta^{1},\beta^{2}$ are given by $\beta^{k}_{j}=(r_{j}-\alpha^{k})_{+}$ for $k=1,2$. Intuitively, the fluid is indifferent between these two dual bases, but, given a realization of $Z$, Offline will prefer one over the other; this causes a discrepancy between the expectations.

Now we turn to the case of any packing problem, the assumption is that we are given two optimal dual solutions $(\alpha^{k},\beta^{k})$, with $\beta^{1}\neq\beta^{2}$. The dual is a minimization problem and $(\alpha^{k},\beta^{k})$ are always dual feasible, thus defining $\beta\coloneqq\beta^{1}-\beta^{2}$ and $\alpha\coloneqq\alpha^{1}-\alpha^{2}$,

[TABLE]

The rest of the proof is reasoning that interchanging expectations $\mathbb{E}[\min_{k=1,2}\{B^{\prime}\alpha^{k}+Z^{\prime}\beta^{k}\}]$ for $\min_{k=1,2}\{B^{\prime}\alpha^{k}+\mathbb{E}[Z]^{\prime}\beta^{k}\}$ induces a $\Omega(\sqrt{T})$ error.

Since the two dual solutions have the same dual value, $B^{\prime}\alpha^{1}+\mu^{\prime}\beta^{1}=B^{\prime}\alpha^{2}+\mu^{\prime}\beta^{2}$, we conclude $B^{\prime}\alpha=-\mu^{\prime}\beta$. We can use this condition to rewrite our bound as

[TABLE]

Since $v(P[\mu])=B^{\prime}\alpha^{k}+\mu^{\prime}\beta^{k}$ for $k=1,2$, we take a random convex combination to obtain

[TABLE]

Now combine the last with our upper bound for $v(P[Z])$ and take expectations to obtain

[TABLE]

Let us define $\xi$ as the normalized vector $Z$, i.e., $\xi\coloneqq\frac{1}{\sqrt{T}}(\mu-Z)$. We conclude that

[TABLE]

Reducing by the standard deviation and applying the Central Limit Theorem, we arrive at a half-normal (also known as folded normal), which has constant expectation. This concludes the desired result.

10 Additional Details and Proofs

10.1 Poisson Process in Discrete Periods

We explain how a continuous time Poisson process can be reduced to our setting. We are given a time horizon $T$, where time $t\in[0,T]$ still denotes time to go and, according to an exponential clock, arrivals occur at some times $t_{1}>t_{2}>\ldots>t_{N}\in[0,T]$, where $N$ is random and corresponds to the total number of arrivals, i.e., $N=\sum_{j\in[n]}Z_{j}(T)$.

Treating times $t_{k}$ as periods, there is one arrival per period. Observe that Offline knows $N$, therefore his Bellman Equation is well defined. Online acts on these discrete periods, i.e., he is event-driven, thus making at most $N$ decisions. Finally, we note that, at some time $t_{k}$, $\mathbb{E}[Z_{j}(t_{k})]=\lambda_{j}t_{k}$ if the process is homogeneous or $\mathbb{E}[Z_{j}(t_{k})]=\int_{0}^{t_{k}}\lambda_{j}(t)dt$ if the process is non-homogeneous In conclusion, Online can compute all the required expectations without knowing $N$, but rather the knowledge of $t_{k}$ and $\lambda(\cdot)$ is enough.

10.2 Bayes Selector Based on Marginal Compensations

A somewhat more powerful oracle is one which, for every time $t$, state $s$ and action $a$, returns estimates of the marginal compensation $\partial R(t,a,s)\cdot\mathds{1}_{Q(t,a,s)}$. This suggests a stronger form of the Bayes selector based on marginal compensations, as summarized in Algorithm 5.

The following result follows directly from Lemma 3.9 and gives a performance guarantee for this algorithm.

Corollary 10.1** **(Regret Of Marginal-Compensation Bayes Selector)

Consider Algorithm 5 with overestimates $\hat{l}(t,a,s)$, i.e., $\hat{l}(t,a,s)\geq\mathbb{E}[\partial R(t,a,s)\mathds{1}_{Q(t,a,s)}]$. If $A^{t}$ denotes the policy’s action at time $t$, then

[TABLE]

10.3 Multi-Secretary Problem

Proof 10.2** **(Proof of Theorem 4.2)

Assume w.l.o.g. that $r_{1}\geq r_{2}\geq\ldots\geq r_{n}$. This one dimensional version can be written as follows.

[TABLE]

The optimal solution to $(P_{t}^{\star})$ is to sort all the arrivals by reward and pick the top ones. The solution to $(P_{t})$ is similar, except that it can be fractional; we saturate the variable $x_{1}$ to $tp_{1}$, then $x_{2}$ to $tp_{2}$, and continue as long as $\sum_{i\leq j}tp_{i}\leq B^{t}$ for some $j$. Define the probability of ‘arrival $j$ or better’ by $\bar{p}_{j}\coloneqq\sum_{i\leq j}p_{i}$. Observe that we can saturate all variables $1,\ldots,j$ iff $t\bar{p}_{j}\leq B^{t}$. The solution to $(P_{t})$ is therefore to pick the largest $j$ such that $t\bar{p}_{j}\leq B^{t}$, then make $X^{t}_{i}=tp_{i}$ for $i\leq j$ and $X^{t}_{j+1}=B^{t}-t\bar{p}_{j}$. When we round this solution according to Algorithm 2, we arrive at the following policy: First, if $B^{t}=0$, end the process. Second (assuming $B^{t}\geq 1$), always accept class $j=1$. Third (assuming $B^{t}\geq 1$), if class $j>1$ arrives, accept if $B^{t}/t\geq\bar{p}_{j}-p_{j}/2$ and reject if $B^{t}/t<\bar{p}_{j}-p_{j}/2$.

Recall that $q(t,b)$ is the probability that Offline is not satisfied with Online’s action at time $t$ if the budget is $b$. We denote $q_{j}(t,b)$ as the probability conditioned on $\theta^{t}=j$. Our aim in the rest of the section is to show that $q_{j}(t,b)$ is summable over $t$.

As we observed before: (1) Offline is not satisfied rejecting a class $j$ iff he accepts all the future arrivals type $j$, i.e., $X^{\star t}_{j}>Z_{j}(t)-1$. (2) Offline is not satisfied accepting class $j$ iff he rejects all future type $j$ arrivals, i.e., $X^{\star t}_{j}<1$. We use the following standard Chernoff bound in [22, Theorem 1.1]. For any $\alpha\in[0,1]$, if $X\sim\text{Bin}(t,\alpha)$:

[TABLE]

We now bound the disagreement probabilities $q_{j}(t,B^{t})$. Take $j$ rejected by Online, i.e., it must be that $j>1$ and $B^{t}/t<\bar{p}_{j}-p_{j}/2$. Since we are rejecting, a compensation is paid only when condition (1) applies, thus $X^{\star t}_{j}=Z_{j}(t)$. By the structure of Offline’s solution, all classes $j^{\prime}\leq j$ are accepted in the last $t$ rounds, i.e., it must be that $X^{\star t}_{j^{\prime}}=Z(t)_{j^{\prime}}$ for all $j^{\prime}\leq j$. We must be in the event $\sum_{j^{\prime}\leq j}Z(t)_{j^{\prime}}\leq B^{t}$. We know that $\sum_{j^{\prime}\leq j}Z(t)_{j^{\prime}}\sim\operatorname{Bin}(t,\bar{p}_{j})$. Since $B^{t}/t<\bar{p}_{j}-p_{j}/2$, the probability of error is:

[TABLE]

Using Eq. 20, it follows that $q_{j}(t,B^{t})\leq e^{-p_{j}^{2}t/2}$.

Now let us consider when $j$ is accepted by Online. A compensation is paid only when $j>1$ and condition (2) applies, thus $X^{\star t}_{j}=0$. Again, by the structure of $X^{\star t}$, necessarily $X^{\star t}_{j^{\prime}}=0$ for $j^{\prime}\geq j$. Therefore we must be in the event $\sum_{j^{\prime}<j}Z(t)_{j^{\prime}}\geq B^{t}$. Recall that $j$ is accepted iff $B^{t}/t\geq\bar{p}_{j}-p_{j}/2=\bar{p}_{j-1}+p_{j}/2$, thus

[TABLE]

This event is also exponentially unlikely. Using Eq. 20, we conclude $q_{j}(t,B^{t})\leq e^{-p_{j}^{2}t/2}$. Overall we can bound the total compensation as:

[TABLE]

Using compensated coupling (Lemma 3.9), we get our result.

Proof 10.3** **(Proof of Corollary 4.3)

By Corollary 3.13, if $A^{t}$ is the action using over-estimates $\hat{q}$, then $\mathbb{E}[\textsc{Reg}]\leq r_{\max}\sum_{t\in[T]}(\mathbb{E}[\hat{q}(t,A^{t},B^{t})]+\Delta^{t})$. Recall that $A^{t}$ is chosen to minimize disagreement, hence, given the condition $\lvert q(t,a,b)-\hat{q}(t,a,b)\rvert\leq\Delta^{t}$, we have $\hat{q}(t,A^{t},B^{t})\leq\min_{a\in\mathcal{A}}q(t,a,B^{t})+\Delta^{t}$. In conclusion,

[TABLE]

We prove that $\min_{a\in\mathcal{A}}q_{j}(t,a,b)\leq e^{-p_{j}^{2}t/2}$ for all $t\in[T],j\in[n],b\in\mathbb{N}$, hence the corollary follows by summing all the terms.

Let us denote $a=1$ the action accept and $a=0$ reject. In the proof of Theorem 4.2 we concluded that the following are over-estimates of the disagreement probabilities $q$:

[TABLE]

Crucially, observe that the term $e^{-p_{j}^{2}t/2}$ is independent of the state $b$. This proves that $\sup_{b\in\mathbb{N}}\min\{q_{j}(t,0,b),q_{j}(t,1,b)\}\leq e^{-p_{j}^{2}t/2}\,\forall\,t\in[T],\,\forall\,j\in\Theta$. The proof is completed.

10.4 Other Arrival Processes

Proof 10.4** **(Proof of Example 4.13)

This follows from an application of [16, Theorem 3.1], which guarantees that, for some constants $c^{\prime},m$ that depend on $P$ only,

[TABLE]

To obtain Eq. 12, we fix $j\in[n]$ and use a union bound taking the worst case in Eq. 22; we let $\nu_{\min}\coloneqq\min_{k\in[n]}\nu_{k}$, $\nu_{\max}\coloneqq\max_{k\in[n]}\nu_{k}$ and set $\delta=\nu_{j}/2\kappa_{j}\nu_{\max}$ in Eq. 22 to obtain the result. The constants are thus $c_{j}=(\nu_{j}/2\kappa_{j}\nu_{\max})^{2}\nu_{\min}/72m$. Finally, we mention that the constants $c^{\prime}$ and $m$ are related to the spectral gap and mixing time of $P$, for details see [16].

Proof 10.5** **(Proof of Example 4.14)

To prove the All-Time Deviation, we use that, from the proof of [21, Lemma 3], $\mathbb{P}(\lvert X-\mathbb{E}[X]\rvert\geq\varepsilon\mathbb{E}[X])\leq 2e^{-\mathbb{E}[X]\varepsilon^{2}/4}$ is valid for any Poisson r.v. $X$ and any $\varepsilon>0$. Now we proceed as in Example 4.13: taking $X_{k}=Z_{k}(t)$ and $\varepsilon=\frac{\mathbb{E}[Z_{j}(t)]}{2\kappa_{j}\mathbb{E}[Z_{k}(t)]}$ we obtain

[TABLE]

Finally, from Eq. 13 we have $\mathbb{E}[Z_{k}(t)]\leq g(t)\mathbb{E}[Z_{j}(t)]$ and from Eq. 14 we have $\mathbb{E}[Z_{j}(t)]\geq g(t)f(t)\log(t)$. From these bounds, we conclude $\mathbb{P}\left[\lvert\lvert Z(t)-\mathbb{E}[Z(t)]\rvert\rvert_{\infty}\geq\frac{\mathbb{E}[Z_{j}(t)]}{2\kappa_{j}}\right]\leq 2ne^{-f(t)\log(t)/8\kappa_{j}^{2}}$ and the existence of constants $\tau_{j},c_{j}$ satisfying the All-Time Deviation follows.

10.5 Proof of Proposition 5.3

We denote $\mathbf{x}\in\mathbb{R}^{nd}$ the vector of the form $\mathbf{x}=(x_{11},x_{21}\ldots,x_{d1},x_{12},\ldots)^{\prime}$, i.e., we concatenate the components $x_{ij}$ by $j$ first. We can write the feasible region of $P[z,b]$ as $\{\mathbf{x}:C\mathbf{x}\leq b,D\mathbf{x}\leq z,\mathbf{x}\geq 0\}$, where $C\in\mathbb{R}^{d\times nd}$ and $D\in\mathbb{R}^{n\times nd}$. It follows from a slight strengthening of [32, Theorem 2.4] that $\lvert\lvert\mathbf{x}^{1}-\mathbf{x}^{2}\rvert\rvert_{\infty}\leq\kappa\lvert\lvert z^{1}-z^{2}\rvert\rvert_{1}$, where

[TABLE]

If we study Eq. 15, denoting $I_{d}$ the $d$-dimensional identity and $1_{d},0_{d}$ $d$-dimensional row vectors of ones and zeros, we can write the matrices $C,D$ as follows. We sketched the multipliers $u_{i},v_{j}$ next to the rows,

[TABLE]

and similarly

[TABLE]

We have two cases: either $u_{i}=0$ for some $i\in[d]$ or $u_{i}\neq 0$ for all $i\in[d]$. On the first case, say w.l.o.g. $u_{1}=0$ and take any $j\in[n]$. Observe that the constraint $\lvert\lvert C^{\prime}u+D^{\prime}v\rvert\rvert_{1}=1$ implies (studying all the components involving $j$) $\sum_{i\in[d]}\lvert u_{i}+v_{j}\rvert\leq 1$. Since $u_{1}=0$, this reads as $\lvert v_{j}\rvert+\sum_{i>1}\lvert u_{i}+v_{j}\rvert\leq 1$, thus $\lvert v_{j}\rvert\leq 1$ as desired.

For the other case we assume $u_{i}\neq 0$ for all $i$, hence $v_{j}=0$ for some $j$, since otherwise we would violate the l.i. restriction on the support. Assume w.l.o.g. $v_{1}=0$ and let us study some $v_{j}$. The constraint $\lvert\lvert C^{\prime}u+D^{\prime}v\rvert\rvert_{1}=1$ implies (looking at the first $n$ components and the components involving $j$) $\sum_{i\in[d]}\lvert u_{i}\rvert+\sum_{i\in[d]}\lvert v_{j}+u_{i}\rvert\leq 1$. By triangle inequality,

[TABLE]

This shows $\lvert v_{j}\rvert\leq 1$ and the proof is complete. $\square$

10.6 Additional Details for Online Stochastic Matching

The stochastic bipartite matching is defined by a set of static nodes $U$, $\lvert U\rvert=d$, and a random set of nodes arriving sequentially. At each time a node $\theta^{t}$ is chosen from a set $V$, $\lvert V\rvert=n$, and we are given its set of neighbors in $U$. We identify the online bipartite matching problem in our framework as follows. The state $S^{t}$ encodes the available nodes from $U$, an action corresponds to matching the arrival $\theta^{t}\in V$ to a neighbor $u\in U$ of $\theta^{t}$ or to discard the arrival. In the latter case we say that it is matched to $u=\varnothing$.

For a graph $G$, we denote the size of its maximum matching as $M(G)\in\mathbb{N}\cup\{0\}$ and $G-(u,v)$ as the usual removal of nodes; in the case $u=\varnothing$, $G-(u,v)=G-v$. Recall that $Q(t,a,s)$ is the event when Offline is not satisfied with action $a$ and $q(t,a,s)=\mathbb{P}[Q(t,a,s)]$. Let us fix an Online policy and define $G_{t}=(L,R)$ as the bipartite graph with nodes $L=S^{t}$ and $R=Z(t)$, i.e., the realization of future arrivals and current state. With the convention $\mathds{1}_{\varnothing}=0$ and $\mathds{1}_{u}=1$ for $u\in U$,

[TABLE]

In words, Offline is satisfied matching $\theta^{t}$ to $u$ if the size of the maximum matching with and without that edge differs by exactly $1$. With this observation, a straightforward application of the compensated coupling Lemma 3.9 yields Lemma 5.4.

Finally, we provide an example for a negative result. Despite the fact that the regret is exactly the number of disagreements and the Bayes Selector minimizes each term, it is not an optimal policy.

Proposition 10.6

The Bayes Selector is sub-optimal for stochastic online bipartite matching.

Proof 10.7

Consider an instance with static nodes $U=\{a,b,c\}$ and four types of online nodes $V=[4]$. Type $1$ matches to $a$ only, $2$ to $a$ and $b$, $3$ to $c$ only, and $4$ to $b$ and $c$. Observe that the only types inducing error are $2$ and $4$.

Assume the arrival at $t=3$ is $\theta^{3}=2$. Matching it to $a$ is an error if arrivals are $\{1,1\},\{1,3\},\{1,4\}$, so the disagreement is $p_{1}^{2}+2p_{1}p_{3}+2p_{1}p_{4}$. Matching it to $b$ is an error if arrivals are $\{4,4\},\{3,4\}$ with disagreement $p_{4}^{2}+2p_{4}p_{3}$. Now assume $p_{4}^{2}+2p_{4}p_{3}=p_{1}^{2}+2p_{1}p_{3}+2p_{1}p_{4}$, so the Bayes selector is indifferent and thus say it matches to $a$.

At $t=2$, there is only an error if $\theta^{2}=4$, in which case matching it to $b$ has disagreement $p_{2}$ and matching it to $c$ disagreement $p_{3}$. In conclusion to the Bayes selector pays $p_{1}^{2}+2p_{1}p_{3}+2p_{1}p_{4}$ in the first stage plus $\min\{p_{2},p_{3}\}$ in the second with probability $p_{4}$.

The strategy that matches at $t=3$ type 2 to $b$ has disagreement $p_{4}^{2}+2p_{4}p_{3}=p_{1}^{2}+2p_{1}p_{3}+2p_{1}p_{4}$, thus lower than the Bayes selector. To see this, note that if we match to $b$ there is no error at $t=2$.

Finally, the equation $p_{4}^{2}+2p_{4}p_{3}=p_{1}^{2}+2p_{1}p_{3}+2p_{1}p_{4}$ is satisfied, e.g., with $p_{1}=p_{4}/2$ and $p_{3}=p_{4}/4$.

11 Additional Details from Numerical Experiments

Competitive is described as follows. For a given horizon $T$, let $K_{j}\coloneqq\lceil p_{j}T\rceil$. We create a bipartite graph $G=(U,V,E)$, where $U$ is the static side and $V$ the online side. The static side contains $B_{i}$ copies of each resource $i$, hence $\lvert U\rvert=\sum_{i\in[d]}B_{i}$. The online side contains $K_{j}$ copies of each type $j$, hence $\lvert V\rvert=\sum_{j\in[n]}K_{j}$. The edge set $E$ is the natural construction where each copy of $j\in[n]$ has edges to all copies of $i\in[d]$ according to the adjacency matrix $A$. The weight $w_{e}$ on edge $e=(u,v)$ is $r_{ij}$ if $u$ is a copy of $i$ and $v$ is a copy of $j$. Finally, define the following matching LP on the graph $G$, where $\lambda_{iljk}$ stands for the $l$-th copy of $i$ and $k$-th copy of $j$

[TABLE]

Let $\lambda^{\star}$ be a solution to this LP. Whenever a type $j$ arrives, Competitive draws $k\in[K_{j}]$ uniformly at random, then takes a vertex $u=il$ incident to $v=jk$ with probability $\lambda^{\star}_{iljk}$ and if $u=il$ is not taken, it matches $v$ to $u$. We note that the process of copying nodes is not superfluous since the analysis of Competitive heavily relies on the fact that the LP is in this form.

Marginal Allocation is described as follows. Let $x$ be a solution of $(P_{T})$ in Eq. 17, i.e., of the fluid LP, and let $f_{i}:[T]\times\{0,\ldots,B_{i}\}\to\mathbb{R}_{\geq 0}$ be some functions specified later. When a type $j$ arrives at $t$ and the current budgets are $B^{t}\in\mathbb{N}^{d}$, Marginal Allocation uses $f_{i}(t,B^{t}_{i})-f_{i}(t,B^{t}_{i}-1)$ as the bid-price for each resource $i\in[d]$: the type is rejected if $r_{ij}<f_{i}(t,B^{t}_{i})-f_{i}(t,B^{t}_{i}-1)$ for all $i\in[d]$ such that $B^{t}_{i}>0$ and otherwise it is matched to $\operatorname{argmax}\{r_{ij}-f_{i}(t,B^{t}_{i})+f_{i}(t,B^{t}_{i}-1):i\in[d],B^{t}_{i}>0\}$. Finally, the functions $f$ are obtained with the following recursion

[TABLE]