A Dominant Strategy Truthful, Deterministic Multi-Armed Bandit Mechanism with Logarithmic Regret
Divya Padmanabhan, Satyanath Bhat, Prabuchandran K.J., Shirish Shevade, and Y. Narahari

TL;DR
This paper introduces a deterministic multi-armed bandit mechanism with a novel concept of -regret that achieves logarithmic regret in sponsored search auctions by leveraging the typical reward separation between agents.
Contribution
It proposes a new -regret framework and a deterministic, incentive-compatible MAB mechanism that attains logarithmic regret, improving over previous methods with higher regret bounds.
Findings
Achieves -regret of O(log T) in sponsored search auctions.
Extends results from single to multiple slot auctions.
Provides a deterministic, incentive-compatible mechanism.
Abstract
Stochastic multi-armed bandit (MAB) mechanisms are widely used in sponsored search auctions, crowdsourcing, online procurement, etc. Existing stochastic MAB mechanisms with a deterministic payment rule, proposed in the literature, necessarily suffer a regret of , where is the number of time steps. This happens because the existing mechanisms consider the worst case scenario where the means of the agents' stochastic rewards are separated by a very small amount that depends on . We make, and, exploit the crucial observation that in most scenarios, the separation between the agents' rewards is rarely a function of . Moreover, in the case that the rewards of the arms are arbitrarily close, the regret contributed by such sub-optimal arms is minimal. Our idea is to allow the center to indicate the resolution, , with which the agents must be distinguished.…
| Babaioff2014 | Our work | |
|---|---|---|
| Loss studied | Regret | -regret |
| Additional parameters | None | : tolerance specified by the planner |
| Mechanism properties | DSIC, deterministic, exploration separated, exploration rounds | DSIC, deterministic, exploration separated, exploration rounds |
| Upper bound on loss | ||
| Lower bound on loss |
| Symbol | Description |
|---|---|
| , | No. of agents and agent set |
| CTR of agent | |
| Valuation of agent for each click | |
| Social welfare when agent is allocated | |
| Click realization of agent at time | |
| Maximum valuation over all agents = | |
| Bid of agent | |
| Bid profile of all agents | |
| Bid profile of all agents except agent | |
| No. of times agent has been selected till time | |
| Allocation at time for bid profile and click realization | |
| Agent with maximum social welfare. Ideally must be allocated at every time step | |
| Social welfare when agent is allocated | |
| Input parameter by center to indicate the level at which the agents must be distinguished | |
| Set of agents whose social welfare is less than away from . These agents do not contribute to -regret. | |
| UCB index corresponding to at time | |
| LCB index corresponding to at time | |
| Empirical CTR of agent estimated from samples up to time | |
| Payment charged to agent if he is allocated a slot at time and he gets a click |
| Symbol | Description |
|---|---|
| No. of slots | |
| Set of slots = | |
| Prominence (Probability with which a user observes an ad at slot given he has observed the ad at slot ) | |
| Probability that an ad at slot is observed | |
| Social welfare when agent is allocated slot | |
| No. of times agent has been alloted slot till time | |
| No. of times agent has been selected till time over all slots | |
| Optimal agent for slot | |
| Social welfare when agent is allocated slot | |
| Set of agents whose social welfare is less than away from . These agents do not contribute to -regret when allocated slot . |
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsAdvanced Bandit Algorithms Research · Auction Theory and Applications · Optimization and Search Problems
∎
11institutetext: † Singapore University of Technology and Design, Singapore
∗ National University of Singapore
‡ Indian Institute of Science (IISc), Bangalore
Dominant Strategy Truthful, Deterministic Multi-armed Bandit Mechanisms with Logarithmic Regret for Sponsored Search Auctions
Divya Padmanabhan*†*
Satyanath Bhat*∗*
Prabuchandran K. J.‡
Shirish Shevade*‡*
Y. Narahari*‡*
(Submitted: December 2018, Revised: May 2020)
Abstract
Stochastic multi-armed bandit (MAB) mechanisms are widely used in sponsored search auctions, crowdsourcing, online procurement, etc. Existing stochastic MAB mechanisms with a deterministic payment rule, proposed in the literature, necessarily suffer a regret of , where is the number of time steps. This happens because the existing mechanisms consider the worst case scenario where the means of the agents’ stochastic rewards are separated by a very small amount that depends on . We make, and, exploit the crucial observation that in most scenarios, the separation between the agents’ rewards is rarely a function of . Moreover, in the case that the rewards of the arms are arbitrarily close, the regret contributed by such sub-optimal arms is minimal. Our idea is to allow the center to indicate the resolution, , with which the agents must be distinguished. This immediately leads us to introduce the notion of -Regret. Using sponsored search auctions as a concrete example (the same idea applies for other applications as well), we propose a dominant strategy incentive compatible (DSIC) and individually rational (IR), deterministic MAB mechanism, based on ideas from the Upper Confidence Bound (UCB) family of MAB algorithms. Remarkably, the proposed mechanism -UCB achieves a -regret of for the case of sponsored search auctions. We first establish the results for single slot sponsored search auctions and then non-trivially extend the results to the case where multiple slots are to be allocated.
Keywords:
Multi-armed bandit mechanismDSIC Deterministic
1 Introduction
Multi-armed bandit (MAB) algorithms Bubeck2012a are now widely used to model and solve problems where decisions are required to be made sequentially at every time step and there is an exploration - exploitation dilemma. This dilemma is the tradeoff that the planner faces in deciding whether to explore arms that may yield higher rewards in the future or exploit the arms that have already yielded high rewards in the past. If the rewards are generated from fixed distributions with unknown parameters, the setting goes by the name stochastic MAB Bubeck2012a . Popular algorithms in the stochastic MAB setting include Upper Confidence Bound (UCB) based algorithms UCBAuer2002 and Thompson Sampling jmlrShipraThompson based algorithms. These algorithms incur regret where is the total number of time steps. MAB algorithms are well studied with several variants Kleinberg2010 ; Bubeck2012b ; Kapoor2018 ; chen2013combinatorial and applications Santiago2017 ; PadmanabhanIJCNN16 ; scott2010modern ; dirkx2018optimizing .
When the arms are controlled by strategic agents, we need to tackle additional challenges. Mechanism design NARAHARI2014 ; NISAN2007 ; Nisan2007jair has been applied in this context, leading to stochastic MAB mechanisms Liu2017 . The design of such mechanisms requires ideas from online learning as well as mechanism design, both of which are increasingly gaining importance in the field of artificial intelligence. An immediate application of stochastic MAB mechanisms is in sponsored search auctions (SSA). In SSA, there are several advertisers who wish to display their ads along with the search results generated in response to a query from an internet user. In the standard model, an advertiser has only one ad to display. We use the terms agent, ad, and advertiser interchangeably. There are two components that are of interest to the planner or the search engine, (1) stochastic component: click through rate (CTR) of the ads or the probability that a displayed ad receives a click (2) strategic component: valuation of the agent for every click that the agent’s ad receives. The search engine would seek to allocate a slot to an ad which has the maximum social welfare (product of click through rate and valuation). However neither the CTRs nor the valuations of the agents are known. This calls for a learning algorithm to learn the stochastic component (CTR) as well as a mechanism to elicit the strategic component (valuation). This problem could become much harder as the agents may manipulate the learning process Babaioff2014 ; Jain2016 to gain higher utilities.
For single slot SSA, it is known that any deterministic MAB mechanism (that is, a MAB mechanism with a deterministic allocation and payment rule) suffers a regret Bubeck2012a ; Feldman2014 of Babaioff2014 . Furthermore, there exists a deterministic MAB mechanism with regret matching the theoretical lower bound Babaioff2014 and also satisfies ex-post truthfulness, the strongest notion of truthfulness (a posteriori to the clicks). When a more relaxed notion of truthfulness is targeted (truthfulness in expectation of the clicks), the regret guarantee improves to babaioff2010truthful . Truthfulness in expectation has also been achieved in Gonen2007 ; Pavlov2009 . The regret can be further improved when randomized mechanisms are used and in fact the regret in this space is babaioff2010truthful ; Jain2018 . However, the high variance that is inevitable to the payments in randomized mechanisms is a serious deterrent to the use of randomized mechanisms. Towards reducing the variance, Ghalme2017 propose a MAB mechanism using Thompson sampling jmlrShipraThompson . However the notion of truthfulness achieved is ‘within period DSIC’ and with high probability. Thus again, only a weaker notion of truthfulness is achieved compared to ex-post truthfulness.
In this work, we observe that the characterization provided by Babaioff et al. Babaioff2014 targets the worst case scenario. In particular, in the lower bound proof of regret of , they consider an example scenario where the actual separation, , between the expected rewards of the arms is a function of . We note that when a similar example () is used with the popular UCB algorithm UCBAuer2002 , the number of pulls of sub-optimal arms could be linear, even in the non-strategic case. Hence, a dependence of on is severely restrictive for the case when the rewards are stochastic, even when the arms are non-strategic. We make the observation that is in most situations independent of . This motivates our main idea in this paper, which is to provide the planner an option to specify a parameter , which is the tolerance or distinguishing level for sub-optimal arms. The understanding is that any arm that is within from the best arm will not cause any additional regret to the planner. For example, the best arm may yield expected reward of while a sub-optimal arm may yield a very close expected reward of . The planner is typically indifferent to such small differences. Traditional exploration-separated schemes end up spending a huge number of exploration rounds in order to distinguish between these two closely separated arms.
**Setting the value of : An Example
** The value of is set by the central planner depending on how well he would like to distinguish between the arms. For example, consider the case where there are two agents. Agent has a CTR and valuation for every click units. Agent has a CTR and a valuation for every click . Agent 1 is the more preferred agent as his expected social welfare is while the expected social welfare for agent 2 is . Then the actual separation between the agents, . But the planner may be indifferent to such a small difference of in expected social welfare. Therefore he would be satisfied with selecting either of the agents. Hence, he should set the parameter to any value greater than .
This notion of tolerance will require an appropriate definition of regret, which we call -regret. Focussing on -regret instead of the usual notion of regret helps us to reduce the number of exploration rounds significantly from to . We propose an exploration separated mechanism based on UCB, which achieves a -regret of . This mechanism can be readily applied in several settings such as SSA, crowdsourcing, and online procurement. For the rest of the paper, however, we use SSA as a running example.
Contributions:
(1) We make the crucial observation that in most MAB scenarios, the separation between the agents’ rewards is rarely a function of (the number of time steps). Moreover, in the case that the rewards of the arms are arbitrarily close, the regret contributed by such sub-optimal arms is negligible. We exploit this observation to allow the center to specify the resolution, , with which the agents must be distinguished. We introduce the notion of -Regret to formalize this regret.
(2) Using sponsored search auctions as a concrete example, we propose a dominant strategy incentive compatible (DSIC) and individually rational (IR) MAB mechanism with a deterministic allocation and payment rule, based on ideas from the UCB family of MAB algorithms. The proposed mechanism -UCB achieves a -regret of for the case of single slot sponsored search auctions. The truthfulness achieved by -UCB is a posteriori to the click realizations and is the strongest form of truthfulness. This loss of would not have been possible otherwise if the traditional notions of regret were used. In particular the number of exploration rounds in -UCB is as opposed to the rounds which were mandatory so far for ensuring a truthful, deterministic mechanism. Thus we now enable the planner to be relieved from this huge number of exploration rounds. We also show that a lower bound on the -regret suffered by any mechanism is .
(3) We non-trivially extend the above results to the case where multiple slots are to be allocated. Here again, our mechanism is DSIC, IR, and achieves a -regret that is .
Our results are generic to stochastic MAB mechanisms and can be applied to other popular applications such as crowdsourcing and online procurement.
2 Relevant Work
In the area of MAB mechanisms, a lot of work has been done in sponsored search auctions. Babaioff et. al.Babaioff2014 provide a characterization of truthful MAB mechanisms, wherein the objective is to maximize social welfare. They introduce the notion of influential rounds. The influential rounds are the rounds where the parameters of reward distributions (CTRs) are learnt. One of the characterizations of truthful deterministic mechanisms is that the allocation must be exploration separated, that is, in such influential rounds, the allocation must not depend on the bids of the agents. The allocation is also required to be point wise monotone. One of the main results of their paper is that any truthful, deterministic MAB mechanism incurs a regret of . In particular, their analysis holds an adversarial nature, as the sub-optimality between the best and second best arm is chosen as if by an adversary, to be proportional to . Such a choice ensures a huge regret for any truthful, deterministic mechanism. They also provide a mechanism which incurs a matching upper bound regret of . Devanur et. al. Devanur2009 concurrently provide similar bounds on the regret when the objective is revenue maximization rather than social welfare maximization.
All the above results pertain to the setting of single slot auctions where there is a single slot for which the agents compete. In the generalization of this setting multiple slots are reserved for ads. This setting is more challenging as every slot is not identical and some slots are more prominent than the others. MAB mechanisms have also been extended to the multiple slot setting Gatti2012 in line with the characterization in Babaioff2014 . Hence, a similar regret of on the social welfare has been attained here as well. Similar results are also stated in the characterisation provided in Akash2012 .
MAB mechanisms have also been proposed in the context of crowdsourcing Biswas2015 . Some of these mechanisms incur a regret of . This is rendered possible due to the specific nature of the problem in hand. In particular, Bhat et. al. BhatAAMAS16 look at divisible tasks. Jain et. al. JainAAMAS16 look at deterministic mechanisms where a block of tasks is allocated to each agent and provide a weaker notion of truthfulness.
The lower bound of both of social welfare regret as well as regret in the revenue of have influenced subsequent research to follow similar assumptions and thereby obtain a similar regret. However, we show in this work that it is indeed possible to design a deterministic mechanism which attains logarithmic regret and is also truthful in the dominant strategy incentive compatible (DSIC) sense Myerson1991 . DSIC, of course, is the most preferred form of truthfulness NARAHARI2014 . This work opens up the possibility for a planner to move away from the worst case scenario to a more realistic scenario. We enable the planner to specify a resolution parameter for distinguishing the arms, introduce the notion of -regret and thereafter propose a mechanism that ensures that the number of exploration rounds and hence the regret suffered is only instead of the expensive available currently in state of the art. We summarize the contrast between our work and the state of the art in Table 1.
3 The Model: Single Slot SSA
We now describe our SSA setting. For ease of reference, our notations are provided in Table 2. Let be the number of agents or arms. We denote the set of arms by . Each of the arms, when pulled, gives rewards from distributions with unknown parameters. We assume here, that the form of the distributions are known but the parameters of the distributions are unknown. In SSA, the rewards of the arms correspond to clicks. The clicks for the advertisements are assumed to be generated from Bernoulli distributions with parameters where is the CTR or probability that advertisement receives a click once observed. The means are unknown.
A click realization represents the click information of every agent at all rounds, that is, if agent received a click in round . In a round , only the click information of the allocated agent is revealed after the completion of the round. Click information of all other unallocated agents is never known to the planner.
The agents also have their valuations for each click they receive. We work in the ‘pay per click’ setting where the agent pays the search engine for each click received. Let the true valuation of agent be for a click. is a private type of agent and is never known to the learner. However the agent is asked to bid his valuation. Let the bid of agent be . We denote by a vector the bid profile of all the agents. The central planner wants to ensure that the agents bid their true valuations, that is must be equal to . Assume that there is a single slot which must be allocated to one of the agents. We denote by the social welfare when agent is allocated a slot, that is, . The social welfare represents the expected valuation of agent per click. If the CTRs of the agents as well as their valuations were known, the planner would have selected the arm with the maximum social welfare, that is, . However neither nor is known to the planner. Assume is the maximum valuation that any agent can have and is common knowledge. The central agent wants to allocate a single slot to one of the ads in such a way that the net social welfare of the allocation is maximized.
A mechanism is a tuple containing an allocation rule and a payment rule . At every time step or round , the allocation rule acts on a bid profile of the agents as well as click realization and allocates the slot to one of the agents, say . Then . Alternatively we denote the indicator variable . The payment rule , where is the payment to be made by agent at time upon receiving a click, when the bids are and for click realization . As stated earlier of the allocated agent alone is observed. Also note that the allocation as well as payments in each round only depends on the click histories till that round.
Let be the arm with the largest social welfare, that is, . We denote the corresponding social welfare as . We denote by the agent chosen at time as a shorthand for . For any given , define the set . denotes the set of all agents separated from the best arm with a social welfare less than . These arms are therefore indistinguishable for the center and they contribute zero to the regret. Note that is a parameter that the center fixes based on the amount in dollars he is willing to tradeoff for choosing sub-optimal arms, given he has only a fixed time horizon to his disposal. To capture this revised and more practical notion of regret, we introduce the metric -regret. Formally,
[TABLE]
The center may not want to invest a huge number of exploration rounds ( in state of the art) to perfectly distinguish the arms that are arbitrarily close. Many a time, the planner may instead be willing to allocate arms that are at most away from the best arm. The center therefore suffers a regret only when an agent with a social welfare greater than away from is chosen. -regret captures this loss.
The goal of our mechanism is to select agents at every round to minimize the -regret.
4 Our Mechanism: -UCB
We are now ready to describe our mechanism -UCB. The idea in -UCB is to explore all the arms in a round-robin fashion for a fixed number of rounds. The number of exploration rounds is fixed based on the desired , specified by the planner. At the end of exploration, with high probability, we are guaranteed that the arms not in are well separated from the best arm with respect to their social welfare estimates. In the exploration rounds, agents need not pay and these rounds are free.
Further on, for all the remaining rounds, the best arm as per the UCB estimate of social welfare is chosen. However in the exploitation rounds, the chosen agent pays an amount for each click he receives. The amount to be paid by the agent is fixed based on variant of the well known Vickrey Clark Grove (VCG) scheme vickrey1961counterspeculation known as weighted VCG NISAN2007 . Note that no learning takes place in these rounds and the UCB, LCB indices do not change thereafter. We present our mechanism in Algorithm 1.
4.1 Properties of -UCB
Next we discuss the properties satisfied by -UCB regarding truthfulness and regret. Before that, we state a few useful definitions which will help in understanding the notion of truthfulness.
At any time step, every agent obtains some utility by participating in the mechanism. This utility is a function of his bid, valuation, bids of other agents and his click realization. Let denote the space of bids of agent . is the bid profile containing bids of all agents except agent . Let denote the space of bids of all agents other than agent . Therefore . We denote by the utility to agent at time when his bid is , his valuation is , the bid profile of the remaining agents is and the click realization is . All agents are assumed to be rational and are interested in maximizing their own utilities.
In our setting the utility to an agent is computed as,
[TABLE]
The idea behind the computation of the utility is as follows. If an agent does not receive an allocation (that is, ), his utility is also zero. He gets a non-zero utility only if he receives an allocation. If he receives an allocation and also a click (), then his utility is the difference between his valuation for the click and the amount he has to pay to the search engine (). If he does not receive a click (), his utility is zero.
Definition 1
Dominant Strategy Incentive Compatible (DSIC) Babaioff2014 : A mechanism is said to be dominant strategy incentive compatible if , .
Note that in the above definition, the truthfulness is demanded a posteriori to even the click realization Gatti2012 . Hence it is the strongest notion of truthfulness. Examples for weaker forms of truthfulness include those which take expectation over click realizations.
Definition 2
Individually Rational (IR): A mechanism is said to be individually rational if , .
Theorem 4.1
-UCB mechanism is dominant strategy incentive compatible (DSIC) and individually rational (IR).
Proof
We analyze the scenarios where an agent bids his true valuation and receives an allocation and also when he does not. We show that in both these scenarios, bidding his true valuation is indeed a best response strategy. We only need to consider the exploitation rounds because in the exploration rounds, every agent is allocated a fixed number of rounds independent of his bids and these rounds are also free for agents.
Case 1:
This implies that when the agent bids his true valuation, he gets an allocation. Therefore for all the other agents . In particular, let agent be such that . The amount to be paid by agent is . If he receives a click then .
Overbid: If agent bids a value , he continues to receive an allocation and his payment is still the same, . Therefore his utility continues to be . Therefore he does not benefit from an overbid.
Underbid: Suppose agent bids a value .
Case a: If is such that , the he fails to get an allocation as . Then the utility to agent is . Therefore he clearly loses his utility by such an underbid.
Case b: Suppose is such that . That is agent bids in such a way that he wins the allocation even with an underbid. Then, if he gets a click, the amount he must pay to the center is . Therefore his utility . He obtains the same utility as a truthful bid and there is no benefit from such an underbid.
Case 2:
This implies that when the agent bids his true valuation, he does not get an allocation. Suppose agent wins the allocation. and .
Truthful bid: Since agent does not win an allocation with a truthful bid, his utility
Overbid: Suppose agent bids in such a way that . We have two sub-cases here.
Case a: If is such that , then agent wins the allocation. So, . If he gets a click, he now has to make a payment . Now his utility . And in particular . Therefore, such an overbid is clearly disadvantageous compared to a truthful bid.
Case b: Suppose . The overbid by agent is not sufficient to make him win the allocation and agent wins the allocation, . The utility of agent , . Therefore there is no advantage for agent by this case of overbid.
Underbid: If agent bids in such a way that , he continues to lose the allocation and therefore his utility,. Since, the utility by an underbid remains the same as a truthful bid, there is clearly no advantage in underbidding.
All the above cases show that our mechanism is DSIC a posteriori to the click realizations. Also, in each of the above cases, note that the utility of an agent , . Therefore, by truthful bidding he never gets a negative utility. This proves that our mechanism is individually rational.
We next discuss the regret incurred by -UCB. We note that the regret analysis we provide differs in spirit from the worst case analysis in Babaioff2014 . The number of exploration rounds in Babaioff2014 is required to be since the separation between the best and second best arm is fixed in an adversarial manner in their analysis. Our analysis does not resort to any adversarial arguments.
In order to prove our -regret results, we will first need to prove several other lemmas.
Lemma 1
*Social Welfare UCB index:
For an agent , we define the social welfare UCB indices for agent as,*
[TABLE]
[TABLE]
Then, .
Proof
Let and denote the UCB and LCB indices for the estimate . Then the events and are identical. So, . An application of Hoeffding bound [hoeffding1963probability ] gives . As per the mechanism . So,
.
Lemma 2
Suppose at time step , . Then , .
Proof
Given that . Therefore,
[TABLE]
Taking square roots on both sides of the above equation yields thereby proving the lemma.
Lemma 3
Suppose . For an agent and time step , let be the event . Define the event , where is the complement of . Then .
Proof
From Lemma 1, the probability of the ‘bad’ event, .
[TABLE]
The last statement follows by summing over all rounds and using the fact that .
Theorem 4.2
Suppose at time step , . Then , with high probability ().
Proof: In Theorem 4.1, we have shown that -UCB is DSIC. Therefore, all the agents bid their valuations truthfully, . Suppose in exploitation round , a sub-optimal arm is pulled. Therefore, . Then one of the following three conditions must have happened.
Condition 1: . This condition implies a drastic overestimate of the sub-optimal arm so that the true social welfare is even below the LCB index . Figure 1 shows this case.
Condition 2: . This implies an underestimate of the optimal arm so that the true social welfare lies above even the UCB index .
**Condition 3: . ** This implies an overlap in the confidence intervals of the optimal and sub-optimal arm. Even though Conditions 1 and 2 are false, still the UCB of sub-optimal arm is greater than the UCB of the optimal arm .
From Figure 3,
If all the three conditions above were false, then,
[TABLE]
This implies that , leading to a contradiction.
As per the statement of the theorem, . Therefore by Lemma 2, . For , . So Condition 3 above does not hold true. So if the sub-optimal arm must have been pulled, only possibilities are for Condition 1 or 2.
[TABLE]
[TABLE]
thereby completing the proof.
We are now ready to state our main result on the incurred regret.
Theorem 4.3
If the -UCB mechanism is executed for a total time horizon of rounds, it achieves an expected -regret of .
Proof
The main idea in the proof is to compute the -regret conditional on two events - and and then to find a bound for these two conditional expectations.
[TABLE]
The last step comes from the fact that Conditions 1 and 2 in the proof of Theorem 4.2 are eliminated as we are given that the event has occurred. After exploration rounds, . From Theorem 4.2, no -regret occurs during exploitation since is true. Therefore the regret is only incurred during the exploration rounds.
We now compute .
[TABLE]
But from Lemma 3.
Putting all the steps together,
[TABLE]
The second term is less than 2 as . This completes the proof.
A consequence of the above theorem is that even if an adversary chooses an arbitrary small gap between the best and second best arm, there is nothing to worry for the planner - if the gap is less than his tolerance , no loss is incurred as opposed to the otherwise loss in Babaioff2014 .
4.2 A Lower Bound for -regret
We will now discuss a lower bound for the -regret incurred by our approach. In particular, we will provide the lower bound for the case where for all and is known. The proof will follow along the lines of the lower bound proof in Bubeck2012a . The same lower bound will also naturally apply to the case of the general strategic version as well, since we our proposed mechanism -UCB is truthful and achieves a matching upper bound.
Let denote the KL divergence between the distributions Bernoulli() and Bernoulli(). Then .
Theorem 4.4
Consider the setting where Suppose an algorithm satisfies for any set of Bernoulli reward distributions and for all arms and . Then for any set of Bernoulli reward distributions we have,
[TABLE]
where , for all .
Proof
We will provide the proof for the case of two agents. The proof for the case follows analogously. Assume that and . Therefore agent is optimal and agent 2 does not belong to . For any , due to the continuity of , we can find such that
[TABLE]
This configuration then corresponds to an alternate setting where the mean of agent 2 is . In this alternate setting, and agent 2 is the unique optimal. For , let,
[TABLE]
It can be verified that (where the expectation is taken over ) and therefore serves as an un-normalized estimate for .
Let denote the following random variable,
[TABLE]
One may verify that by applying a change of measure. We will now show that as . This is due to the following:
[TABLE]
Therefore, setting , and applying Markov inequality we get,
[TABLE]
The last step arises as a consequence of and agent 1 is sub-optimal for the setting where agent 2 has the mean reward of .
We will finally show that .
[TABLE]
Note that and . Therefore by an application of the strong law of large numbers, we have
[TABLE]
Since , we must have as well. Applying Markov inequality again, we get,
[TABLE]
The last step is obtained by applying Equation 8. This completes the proof. Note the key difference between our proof and Bubeck2012a lies in Equation 8. Our RHS in Equation 8 is necessary to ensure that in the alternate scenario agent is sub-optimal.
Remark 1
The lower bound for the expected -regret Theorem 4.4 is quite similar to the lower bound for the regret of the UCB algorithm in Bubeck2012a . The difference is that the KL divergence term in the bound is also a function of the parameter . Intuitively instead of considering the KL divergence between , we give an allowance of for the optimal agent.
5 Extension to Multi-Slot SSA
In the previous sections, we assumed that there was a single slot for which the advertisers were competing. We now look at a more general setting where there are slots to be allocated to the agents. As before, each advertiser has exactly one ad for display and the CTR for advertisement is denoted by . Recall that in the case of single slot auctions, the CTR exactly denoted the probability with which an ad received a click. However in the generalized setting of multi-slot auctions, an additional parameter comes into play while computing the click probability due to which the problem becomes much harder gatti2015truthful .
Each position or slot is associated with a parameter called ‘prominence’. denotes the probability with which a user observes an ad at slot given he has observed the ad at slot . In order to understand the need for this parameter, a useful scenario to imagine is the listing of web-pages in Google for a query. There are two phases that one can think of once the listing of pages or results have appeared.
Phase 1: This is the phase where a user scans through the pages listed. A page listed higher up in the ranking (say second from the top) has more chances of being observed by a user rather than a page that is far below in the ranking (say fifth from the top). , for instance, denotes the probability that a user observes the fifth page, given he has observed the fourth page. Coming back to sponsored ads, we assume that , that is, the ad listed in the first slot is surely observed. We denote by the probability that an ad at slot is observed. is computed as,
[TABLE]
This modeling assumption for is known as position dependent cascade model.
Phase 2: After having scanned through the list, the user decides to click one or more of the shown ads. In the multi-slot setting Gatti2012 , it is assumed that multiple ads in a listing may receive clicks. The probability that ad receives a click when shown at slot = .
We assume that , are known to the planner a-priori. The problem of learning these parameters along with the CTR is much harder in the presence of strategic agents. Therefore, in this section, we work with the assumption that the s and hence s are known. In Section 6.2, we give pointers for design of mechanisms where the s are unknown.
The above modeling assumptions are as per standard conventions gatti2015truthful . In the multi-slot setting, the allocation is given to multiple agents at every time step. We denote by , the allocation at time for bids and click realization . The cardinality of the allocated set . We also use the notation to denote the allocation to agent at time is slot , for the bid profile , click realization . If an agent is not allocated any of the slots at time , we say .
We denote by the social welfare of agent , when he is given slot . is the expected valuation that agent receives when he is given slot and is computed as,
[TABLE]
For ease of reference, the additional relevant parameters for the multi-slot setting are provided in Table 3.
Having described the multi-slot setting, we now analyze the scenario from the view point of the search engine or central planner. In the ideal scenario, the planner would like to allot the ads exactly to the top agents with the largest social welfare. This use case has been studied in the literature Gatti2012 and exploration separated mechanisms with regret of have been proposed. Various possible allocations are explored for time steps for every agent after which the allocation algorithm is guaranteed to converge to the ideal allocation with high probability. As in the single slot case, exploration rounds are required to distinguish all the agents perfectly from each other, when there are agents whose social welfare values are arbitrarily close.
However, a much more practical problem of interest is to study and design mechanisms when the search engine is indifferent to a gap in in social welfare for every slot. We observe that in cases where the agents are well-separated, exploration rounds are not required. In fact, exploration rounds are sufficient to converge to an allocation that is well within the requirements of the search engine.
Having explained the problem, we now formalize the notions of separatedness in this setting. Let be the best agents in terms of their single slot social welfare values, that is, . Let . The ideal solution would be to allocate agent the slot . This allocation would yield the largest social welfare but in the worst case, when the agents’ social welfares are separated by a function of , converging to this optimal allocation would require exploration rounds Gatti2012 . Instead, for a prescribed value of fixed by the search engine, define the set,
[TABLE]
is the set of all agents whose social welfare is at most away from the agent ( who should have ideally been given slot ). The planner is indifferent to the regret contributed by the agents in , if any of them are allotted slot . Hence we define the multi-slot -regret metric as,
[TABLE]
The -UCB mechanism for the multi-slot SSA is given in Algorithm 2.
We analyze the regret and truthfulness of Algorithm 2. The lemmas and theorems for establishing the results for the multi-slot setting are similar to the single slot setting, however there are subtle differences in proving many of the results. We will highlight them as and when necessary.
Theorem 5.1
In the multi-slot setting - is Dominant Strategy Incentive Compatible (DSIC) and Individually Rational (IR).
Proof
The mechanism is an implementation of the weighted VCG scheme (with the weights for each agent and is hence DSIC and IR.
Lemma 4
For an agent and slot , the click through rate UCB indices for agent ,
[TABLE]
[TABLE]
satisfy
Proof
At every time step, we observe samples corresponding to the clicks of the allocated ads. These samples also encompass slot specific information which must be accounted for in the computation of empirical mean as well as UCB index for . For an agent , let the random variable denote whether ad receives a click at slot . Therefore is a Bernoulli random variable with bias .
We obtain a sample of when ad is allocated slot . However it is the samples from that gives us an unbiased estimator for . Therefore, the random variable of interest is the Bernoulli random variable,
[TABLE]
is bounded in and is . Also,
[TABLE]
Consider the scenario where, for an ad , a single sample click is available from each slot. Let denote this sample of . Assume are all independent and . . Now,
[TABLE]
In order to tighten the above bound on the right hand side, one must find appropriate which minimizes . Setting where achieves the minimum value. Therefore,
[TABLE]
In order to obtain a confidence on , must be set so that . Therefore, . In the above analysis we assumed that from each slot, one sample was available. When we have a total of independent samples for ad , with samples for slot at any time , and therefore , completing the proof.
A noteworthy feature of our estimates is the following. An allocation of an ad in a slot yields a sample for the computation of not only , but also for for all slots . This is because is known to the planner a-priori. Therefore note that, the number of allocations that ad receives till time , is the sum of the number of allocations that agent receives irrespective of the slot or inclusive of all the slots.
Lemma 5
For an agent and slot , the social welfare UCB indices for agent ,
[TABLE]
[TABLE]
satisfy
Proof
The proof idea is similar to Lemma 1.
Lemma 6
Suppose at time step , . Then and ,
Proof
The proof is similar to Lemma 2.
Lemma 7
For an agent , slot and time , let be the event . Define the event . Then .
Proof: The proof has some subtle differences from Lemma 3 because in the multi-slot extension, the events are not independent across the slots.
Observation: If an element from the set of outcomes is such that , then . This is because, for any two slots and ,
[TABLE]
Therefore . From Lemma 5,
. Hence,
[TABLE]
Theorem 5.2
Suppose at time , . Then , with high probability ().
Proof: Suppose at time where there exists some such that . (Note that this statement does not arise from any assumptions on the allocation, for instance, that agent is given slot . This is the major difference from Theorem 4.2). But the relation between the true social welfare values of these agents is . Then one of the following three conditions must have occurred, like in proof of Theorem 4.2.
Condition 1: . This condition implies a drastic overestimate of the sub-optimal arm so that the true mean social welfare is even below the LCB index . The figure below captures this condition.
Condition 2: . This implies an underestimate of the optimal arm so that the true mean social welfare lies above even the UCB index . See Figure 5 below.
Condition 3: . This implies an overlap in the confidence intervals of the optimal and sub-optimal arm. Even if, Conditions 1 and 2 are false, still the UCB of sub-optimal arm is greater than the UCB of the optimal arm .
From the figure, . If all the three conditions above were false, then,
[TABLE]
As per the statement of the theorem, . Therefore by Lemma 6, . For agent , . Therefore, Condition 3 above does not hold true. So,
[TABLE]
[TABLE]
Theorem 5.3
If the -UCB mechanism is executed in the multiple slot scenario for a total time horizon of rounds, it achieves an expected -regret of .
Proof
The proof idea has some subtle differences from the proof of Theorem 4.3. As before, we first compute the expected -regret conditional on . For the exploration rounds, the mechanism obtains a regret of .
[TABLE]
We will now show that the second term above evaluates to zero. For any , the cardinality of is at least . This is because for all above in the ranking of agents (), as . Therefore there are at least agents in . Also as . Therefore . While allocating slot , at least one of the agents in must be free. This is by the pigeonhole principle. Now if the allocated agent for slot , , one of the following two cases occur.
Case 1: The ideal agents for all the previous slots have already been allocated before the allocation of slot . This means that has not been allocated yet. Also, . Since is true and , the above event cannot occur (by Theorem 5.2).
Case 2: The agent has already been allocated to some other slot before the allocation of slot has begun. Therefore there is some agent with a larger social welfare value, who has still not been allocated. That is, . Given that . Therefore we can deduce that . This is because,
[TABLE]
The last line in the above implications is true as . But . Then the inequality is also true due to the way the slot specific UCB indices are computed. From Theorem 5.2 for slot , we find that . Again this cannot happen as is true and . Therefore we get that .
Also, from Lemma 7.
Putting all the steps together,
[TABLE]
The simplification in the second line is because . In the last line we use the fact that . This completes the proof.
6 Extensions to Other Variants of Multi-slot SSA
In this section, we look at other variants in the multi-slot SSA setting and discuss how our mechanism can be adapted to such settings.
6.1 Position and Ad Dependent Cascade Model
We have explained our algorithm and performed the analysis for the position dependent cascade model for SSA where the function is characterized by Equation 11 and is known to the planner a-priori. A more general model would be one where the function may also depend on the ad displayed at position . Our model can also be used in such scenarios and the same analysis will hold.
6.2 Handling the Case of Unknown
We have assumed that the functions s are known to the planner a-priori. Now suppose that the s are required to be learnt. The same allocation scheme as in Algorithm 2 may be used. However the computation of the proposed payment scheme in Algorithm 2 is not feasible as the payments use s, which are unknown.
In order to handle such a scenario, we must obtain estimates for first. It is known that, the parameter for the first slot, . Only need to be estimated. We will first describe a mechanism which relies on an arbitrary learning algorithm to provide estimates . Thereafter we will remark on the possible learning schemes.
Proposition 1
Suppose we have a learning scheme that gives us estimates such that, and . Let .
We propose a weighted VCG mechanism [NISAN2007 ] which is known to be DSIC truthful and is also IR. Suppose the private valuation of agent for a click is . Let be an outcome of the allocation. if ad is alloted slot and zero otherwise. The valuation function of agent in this case is,
[TABLE]
Define a weight vector for every agent . has weights corresponding to agent and slot such that, . is the UCB index corresponding to the CTR of ad , computed after the fixed number of exploration rounds as in Algorithm 2. However, in this scenario, the UCB index is constructed using samples of the clicks from allocation in the first slot alone.
Our weighted VCG mechanism is described in Figure 7. The mechanism uses the allocation,
[TABLE]
But note that this allocation rule boils down to the same allocation used in Algorithm 2. This is due to the fact that the estimates monotonically decrease with . The procedure for obtaining the allocation is the following. We sort the agents based on and allocate the slots to the best agents. Therefore, the allocation rule is independent of the s and is equivalent to,
[TABLE]
The expected payment to be made by agent when allocated a slot is,
[TABLE]
The above is the externality based payment prescribed by weighted VCG. However since we adopt the pay per click scheme,
[TABLE]
Therefore, the computation of the payments is also feasible now. The above mentioned weighted VCG scheme is DSIC truthful and IR. The proof follows from the standard weighted VCG scheme where the weights are as defined as above. We now remark on the -regret of the mechanism.
6.2.1 Remarks on Learning and Computation of -regret
In the above mechanism we have assumed, that the estimates satisfy Proposition 1. The allocation scheme described above ultimately does not rely on these estimates, although the weights use it. The mechanism therefore uses the estimates only in the payment rule. We now make an important observation here.
Observation: When any set of estimates , satisfying Proposition 1 is used in the mechanism above, the mechanism is DSIC truthful, IR and suffers only logarithmic -regret.
The reason is that the mechanism is an instance of weighted VCG mechanism and therefore is DSIC truthful and IR, with any estimate for the s. As far as the -regret in social welfare is concerned, the allocation rule determines it. The allocation rule used turns out to be identical to the allocation rule used where is known and is independent of the estimates. Note that it is now possible to minimise regret in payments by choosing estimates that maximise the payments and also satisfy the constraints in Proposition 1. This will lead to a constrained optimization problem which can be solved. However the current work focuses on minimizing -regret in social welfare and therefore the problem of minimising regret in payments is still open.
7 Conclusion
We have studied the more practical use case in MAB mechanisms where a planner has the option to specify a tolerance level for sub-optimal arms. All the papers in the literature on MAB mechanisms propose schemes to target the worst case scenario where the arms are arbitrarily close. Therefore they prescribe investing a huge number of exploration rounds () to perfectly distinguish the arms. However, the planner may not want to perfectly distinguish arms that are arbitrarily close. Many a time, the planner may instead be willing to allocate arms that are at most away from the best arm. The state of the art does not permit this flexibility to the planner. Towards providing such a flexibility to the planner, we have, for the first time, introduced a new notion of regret called -regret. When arms that are less than away from the best arm are selected, the -regret incurred is zero. Only arms more than away from the best arm contribute to the -regret.
From the above perspective, we have revisited the application of MAB mechanisms in sponsored search auctions. First we analysed the single slot SSA setting and proposed a deterministic, exploration separated MAB mechanism called -UCB. We showed that -UCB is DSIC truthful, IR and achieves a -regret of . Next we studied the more challenging setting of multi-slot SSA. In particular, we adopted the cascade model and adapted -UCB to this setting, first with the assumption that the prominence parameters are known. Here too, we have shown that the mechanism is DSIC truthful, IR and achieves a -regret of . We finally adapt the mechanism to the general multi-slot SSA setting where neither the CTRs nor the prominences are known. Here too our deterministic, exploration separated mechanism is DSIC truthful, IR and suffers a -regret of . The other mechanisms in literature for this setting are not able to obtain all these desirable properties that our mechanism achieves. They either compromise on the truthfulness, satisfying a weaker notion (truthfulness in expectation) or are forced to resort to randomness in the mechanism.
Our results are generic and apply equally well to several other applications where MAB mechanisms have been used.
The reference list from the paper itself. Each links out to its DOI / PubMed record.
- 1[1] Shipra Agrawal and Navin Goyal. Analysis of thompson sampling for the multi-armed bandit problem. In COLT , pages 39.1–39.26, 2012.
- 2[2] Peter Auer, Nicolo Cesa-Bianchi, and Paul Fischer. Finite-time analysis of the multiarmed bandit problem. Machine learning , 47(2-3):235–256, 2002.
- 3[3] Moshe Babaioff, Robert D. Kleinberg, and Aleksandrs Slivkins. Truthful mechanisms with implicit payment computation. In Proceedings of the Eleventh ACM Conference on Electronic Commerce (EC’10) , pages 43–52. ACM, 2010.
- 4[4] Moshe Babaioff, Yogeshwer Sharma, and Aleksandrs Slivkins. Characterizing truthful multi-armed bandit mechanisms. SIAM Journal on Computing , 43(1):194–230, 2014.
- 5[5] Satyanath Bhat, Divya Padmanabhan, Shweta Jain, and Yadati Narahari. A truthful mechanism with biparameter learning for online crowdsourcing: (extended abstract). In Proceedings of the 2016 International Conference on Autonomous Agents & Multiagent Systems (AAMAS’16), Singapore, May 9-13, 2016 , pages 1385–1386, 2016.
- 6[6] Arpita Biswas, Shweta Jain, Debmalya Mandal, and Y Narahari. A truthful budget feasible multi-armed bandit mechanism for crowdsourcing time critical tasks. In Proceedings of the 2015 International Conference on Autonomous Agents and Multiagent Systems (AAMAS’15) , pages 1101–1109, 2015.
- 7[7] Sébastien Bubeck and Nicolò Cesa-Bianchi. Regret analysis of stochastic and nonstochastic multi-armed bandit problems. Foundations and Trends in Machine Learning , 5(1):1–122, 2012.
- 8[8] Sébastien Bubeck, Nicolò Cesa-Bianchi, and Gábor Lugosi. Bandits with heavy tail. IEEE Transactions on Information Theory , 59(11):7711–7717, 2013.
