Managing Risk of Bidding in Display Advertising

TL;DR

This paper introduces risk-aware bidding strategies for display advertising that explicitly model uncertainty in user behavior and auction competition, leading to significant profit improvements.

Contribution

It develops a novel risk modeling approach using value at risk from finance, creating bidding strategies that balance expected return and risk in RTB campaigns.

Findings

Profit gains of 15.4% in offline experiments

Up to 17.5% profit increase in online A/B tests

Effective risk-aware strategies outperform traditional methods

Abstract

In this paper, we deal with the uncertainty of bidding for display advertising. Similar to the financial market trading, real-time bidding (RTB) based display advertising employs an auction mechanism to automate the impression level media buying; and running a campaign is no different than an investment of acquiring new customers in return for obtaining additional converted sales. Thus, how to optimally bid on an ad impression to drive the profit and return-on-investment becomes essential. However, the large randomness of the user behaviors and the cost uncertainty caused by the auction competition may result in a significant risk from the campaign performance estimation. In this paper, we explicitly model the uncertainty of user click-through rate estimation and auction competition to capture the risk. We borrow an idea from finance and derive the value at risk for each ad display opportunity. Our formulation results in two risk-aware bidding strategies that penalize risky ad impressions and focus more on the ones with higher expected return and lower risk. The empirical study on real-world data demonstrates the effectiveness of our proposed risk-aware bidding strategies: yielding profit gains of 15.4% in offline experiments and up to 17.5% in an online A/B test on a commercial RTB platform over the widely applied bidding strategies.

Tables

Table 1. Table 1: Notations and descriptions

Notation	Description
$𝒙$	The features of bid request.
$𝒘$	The weights of features in CTR estimation function.
$y$	The true label of user response.
$\hat{y}$	The estimated label of user response.
$𝝁$	The mean vector of $𝒘$.
$𝑺$	The covariance matrix of $𝒘$.
$q_i$	The precision (reciprocal of variance) of $w_i$.
$v$	The pre-defined value of positive user response.
$b$	The bid price determined by bidding strategy.
$z$	The market price determined by second-price auction.
$\alpha$	The coefficient balancing utility and risk.
$R$	The utility of advertiser.
$p\left(\right)$	The probability density function.
$P\left(\right)$	The probability mass function.
$\sigma \left(\right)$	The sigmoid function.

Equations

$$\hat{y}=\sigma ({\mathbf{w}}^T\mathbf{x})=\frac{1}{1+e^{-{\mathbf{w}}^T\mathbf{x}}},$$

$$p(y|\mathbf{x},\mathbf{w})=\sigma ({\mathbf{w}}^T\mathbf{x}{)}^y(1-\sigma ({\mathbf{w}}^T\mathbf{x}){)}^{(1-y)}.$$

$$p(y|\mathbf{x})={\int }_{\mathbf{w}}p(y|\mathbf{x},\mathbf{w})p(\mathbf{w})d\mathbf{w}.$$

$$p(\mathbf{w}|\mathbf{x},y)=\frac{p(y|\mathbf{x},\mathbf{w})p(\mathbf{w})}{{\int }_{{\mathbf{w}}^{\prime }}p(y|\mathbf{x},{\mathbf{w}}^{\prime })p({\mathbf{w}}^{\prime })d{\mathbf{w}}^{\prime }}$$

$$\propto$$

$$p(\mathbf{w})=N(\mathbf{w}|{\mathbf{\mu }}_{\text{new}},{\mathbf{S}}_{\text{new}}),$$

$${\mathbf{\mu }}_{\text{new}}=$$

$$-\frac{1}{2}\sum _i{q}_{\text{old},i}(w_i-{\mu }_{\text{old},i}{)}^2+\text{const},$$

$${\mu }_{\text{new},i}\leftarrow$$

$$\displaystyle~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}-q_{\text{old},i}(\mu_{\text{new},i}-\mu_{\text{old},i})\Big{)}$$

$${\mathbf{S}}_{\text{new}}^{-1}=$$

$$q_{\text{new},i}={\mathbf{S}}_{\text{new},i,i}^{-1}=q_{\text{old},i}+\sigma ({\mathbf{\mu }}^T\mathbf{x})(1-\sigma ({\mathbf{\mu }}^T\mathbf{x}))x_i^2.$$

$$\hat{y}=P(y=1|\mathbf{x})=\sigma ({\mathbf{w}}^T\mathbf{x}),w_i\sim N({\mu }_i,q_i^{-1}),$$

$$\displaystyle\hat{y}=\sigma(a),~{}~{}~{}a\sim N\Big{(}\sum_{i}\mu_{i}x_{i},\sum_{i}q_{i}^{-1}x_{i}\Big{)}~{}.$$

$$\displaystyle p_{y}(y)=p_{x}(g^{-1}(y))\Big{|}\frac{\partial g^{-1}(y)}{\partial y}\Big{|}~{}.$$

$$p_{\hat{y}}(\hat{y})=$$

$$r=v\cdot \hat{y},p_r(r)=p_{\hat{y}}(\hat{y})/v.$$

$$b^{\ast }$$

$$=\underset{b}{\mathrm{argmax}}{\int }_{\hat{y}}{\int }_{z=0}^b(v\cdot \hat{y}-z)p_z(z)dz\cdot p_{\hat{y}}(\hat{y})d\hat{y}.$$

$$\frac{\partial \mathbb{E}[R(b)]}{\partial b}$$

$$=p_z(b){\int }_{\hat{y}}(v\cdot \hat{y}-b)p_{\hat{y}}(\hat{y})d\hat{y}=0$$

$$\Rightarrow b^{\ast }$$

$$P(R(b)<0)=$$

$$=$$

$$P(r<\mu -\alpha \sigma )<\frac{1}{1+{\alpha }^2},\alpha >0,$$

$$P(r>\mu -\alpha \sigma )<\frac{1}{1+{\alpha }^2},\alpha <0.$$

$$\tilde{r}=\mathbb{E}[r]-\alpha \mathrm{Std}[r],$$

$$b_{\text{VaR}}(r)=\tilde{r}=v\cdot (\mathbb{E}[\hat{y}]-\alpha \mathrm{Std}[\hat{y}]),$$

$$R(b)=\{\begin{array}{ll}0& b\le z\text{(lose)}\\ v\cdot \hat{y}-z& b>z\text{(win)}\end{array}.$$

$$\tilde{R}(b)=\mathbb{E}[R(b)]-\alpha \mathrm{Std}[R(b)],$$

$$b_{\text{RMP}}(R(b))=\underset{b}{\mathrm{argmax}}\mathbb{E}[R(b)]-\alpha \mathrm{Std}[R(b)].$$

$$b_{\text{LR}}=\varphi \cdot v\cdot \hat{y},$$

$$\mathbb{E}[R(b)]=v\cdot \mathbb{E}[\hat{y}]\cdot z_0-z_1,$$

$$\mathbb{E}[R(b{)}^2]=v^2\mathbb{E}[{\hat{y}}^2]z_0-2v\mathbb{E}[\hat{y}]z_1+z_2.$$

Full text

††conference: WSDM 2017, February 06-10, 2017, Cambridge, United Kingdom

1 Introduction

Display advertising has become a significant battlefield for big data [33]. As the advertising transactions are aggregated across websites in real time, the display advertising industry has a unique opportunity to understand the internet traffic, user behaviors, and online transactions. In this paper, we have used datasets from a leading DSP (Demand Side Platform) in China, iPinYou, which processes up to 18 billion ad impressions per day [3]. The advertising platform YOYI, which has deployed our proposed algorithms in this paper, currently handles 5 billion ad transactions daily. Moreover, Fikisu DSP claims to process 32 billion ad impressions daily [2]; Turn reports to handle 2.5 million per second in the peak time [29]. To get the picture of the scale, the New York Stock Exchange trades around 12 billion shares daily [1], while the Shanghai Stock Exchange trades about 14 billion shares daily [4]. It is fair to say that the transaction volume from display advertising has already surpassed that of the financial market.

To further compare, similar to the financial market, ad impression trades are also automated largely by auction mechanisms — while the financial market uses a double auction to create bid and ask quotes [6], RTB (Real-time Bidding) display advertising adopts the second-price auction to gather bid quotes from advertisers once an impression is being generated [38, 23]. As it is possible now to track user actions resulted from an online campaign, advertising optimization becomes more resembling to that of the financial market trading and tends to be driven by the marketing profit and return-on-investment (ROI). That is, there is explicit and measurable campaign goal of acquiring new users from the campaign in order to obtain additional sales from the acquired users. Thus, how to properly bid an ad impression to drive the profit and ROI becomes essential to performance-driven campaigns [25, 39].

Bidding strategies are normally built by estimating the utility of each ad impression, which is commonly done by predicting the underlying user’s click-through rate (CTR) or conversion rate (CVR) [17]. Existing solutions for predicting CTR range from linear models [13, 21, 15, 17, 26] and gradient boosting decision trees (GBDT) [31] to factorization machines [24]. All of them aim to return a predicted CTR value of the given ad impression. However, we never know whether such a point estimation is confident enough. The true underlying CTR may heavily deviate from the predicted value, given the significant uncertainty of the underlying user behavior. Such utility fluctuation, along with the cost uncertainty caused by the auction competitions [5], results in a significant risk of the campaign profit estimation which should not be ignored.

In this paper, we depart from conventional CTR point estimations and explicitly model the CTR distribution to capture the uncertainty (risk) of the utility measure for each potential ad impression. Our idea is inspired from finance about value at risk: a statistical technique used to measure and quantify the level of financial risk with an investment [19]. The value at risk of an ad display opportunity is generally defined as the lower bound of the ad impression value guaranteed by a specific probability (risk). With the measured CTR prediction and market price risks, we propose two practical risk-aware bidding strategies to handle the uncertainty from both utility estimation and market competition. Our methods are evaluated on two large-scale real-world ad datasets. With campaign profit as the key performance indicator (KPI), we find that our risk-averse bidding strategies, which penalize the bids with high uncertain CTR (or profit) and award the confident ones, yield a 15.4% profit gain over a linear bidding strategy with a conventional logistic regression CTR estimator [25] and largely outperform the risk-neutral and risk-seeking bidding.

Furthermore, we have deployed the risk-averse bidding strategies on YOYI’s DSP and conducted a 7-day online A/B test. In the live test, our proposed risk-aware bidding strategies bring 17.5% higher campaign profit and 61.5% higher CTR over the conventional ones by effectively saving money on uncertain and low-value opportunities, which verifies the practical effectiveness of our solutions of managing advertising bidding risk.

The rest of this paper is organized as follows. We discuss the related work in Section 2. Our CTR distribution modeling is provided in Section 3. In Section 4, the concept of value at risk and the derived risk-aware bidding strategies are discussed. Offline and online experimental results are provided in Sections 5 and 6, respectively. We finally conclude this paper and discuss our future work in Section 7.

2 Related Work

User Response Prediction. Predicting the probability of a specific user response, e.g., CTR and CVR, is a key function for performance-driven online advertising [13, 21, 15]. The applied CTR estimation models today are mostly linear. Logistic regression is the most widely used model, normally trained by stochastic gradient descent (SGD) [17, 26]. The authors in [21] proposed to use an online learning algorithm called follow-the-regularized-leader (FTRL) to train logistic regression from the streaming data. The model successfully bypasses the learning rate update problem in SGD and it empirically works effectively. Bayesian probit regression [13] is another linear model for online learning where the feature weights are modeled with a distribution and the model learning is via updating the weight posterior. Binary naive Bayes [14] is also a popular linear model, by assuming the features are conditionally independent.

Linear models are simple and effective in learning, but may fail to capture the interactions between the assumed (conditionally) independent raw features [13]. By contrast, non-linear models are capable of learning feature interactions in various ways and could potentially improve prediction performance [24, 31]. Gradient boosting decision trees (GBDT) [31, 15] are a straightforward non-linear model to capture feature interactions. Moreover, latent factor models, particularly factorization machines (FMs) [24], map each binary feature into a low dimensional continuous space, and the feature interaction is automatically explored via vector inner product.

Real-Time Bidding Strategies. The emergence of ad exchanges for display advertising in 2009 [23] provides automatic trading mechanism for advertisers to buy media inventory in impression level and determine the acceptable price via second price auction [38].

The authors in [12] proposed an algorithm that made dynamic bidding decisions to achieve an optimal delivery with the budget constraint. In [8], the bid price from each campaign was adjusted by the publisher or the supply-side platform in real time and the goal was to maximize the publisher side profit. Borrowing the idea of the optimal truth-telling bidding in sponsored search [10], a basic bidding strategy is to bid the estimated true value for each ad impression. For performance-driven campaigns, the predefined true value is normally based on the economical value of the user actions, such as a click or a conversion. The expected true value for a specific impression is estimated as action value multiplied by action rate [17, 8]. However, the truth-telling bidding strategy is optimal only when the budget and auction volume are not considered. With the campaign budget and lifetime auction volume constraints, the optimal bidding strategy may not be truth-telling. Extending from the truth-telling bidding strategy, the authors in [25] proposed the generalized bidding function with a linear relationship to the predicted CTR for each ad impression being auctioned. Compared to [25], the authors in [40] proposed a functional optimization framework to directly optimize the bidding function, where the derived function showed that an optimal bidding function could be non-linear w.r.t. predicted CTR. The non-linearity is closely related to the distribution of market price [5]. Extending to multiple campaign bid optimization task, the authors in [39] further devised linear and non-linear bidding functions from the functional optimization framework. Extending from previous strategies, our work introduces a new factor, i.e. the standard deviation of predicted CTR, to be considered by bidding function in addition to predicted CTR.

Risk Management and Applications. Risk is a consequence of action taken in spite of uncertainty [22]. The objective of risk management is to assure uncertainty does not to-some-extent deflect the business from its goals [9].

Modern portfolio theory (MPT) [20] originates from modeling uncertainty of the return of combinations of multiple financial assets. It presents a quantitative method to measure such uncertainty (or risk) and embeds it into the decision making of investment [16]. In MPT, the variance of the return of each asset is modeled as its risk. Then the risk and expected return of a portfolio of invested assets are quantified by the fund allocation, the mean return of the assets and their covariance matrix [20, 16]. MPT utilizes the mean-variance analysis to make an investment portfolio for any tradeoff between the risk and the expected return, or w.r.t. a reference investment such as bank rates [28]. With such advantages, MPT has been adopted in almost everywhere of financial investment [11].

Recently, the ideas of risk management has been introduced to information retrieval, such as document ranking in web search [34, 35] and diversification in top-N recommendation [30, 43], to improve the model robustness or catch the users’ satisfaction on uncertainty psychologically. In the area of recommender system, bandits solutions [42] have been proposed to model confidence interval to balance exploration and exploitation in a risk-seeking fashion.

Computational advertising is associated with a certain level of deficit risk, particularly for performance-driven campaigns as the goal is to acquire new users and gain more sales from them. The risk comes from the dynamics of the market and the user online behaviors [32]. The authors in [39] proposed to measure campaign-level risk and return in a special case of arbitrage between CPM and CPA. Compared to [39], our work focuses on single campaign optimization, and our risk is modeled from the uncertainty of user response and market competition at impression-level. Generally, our work borrows the concept of value at risk from finance to derive risk-aware bidding strategies intending to reasonably allocate budget between uncertain impressions and confident impressions and achieve a campaign-level profit gain, which is unlike in finance where risk control is only for balancing return and risk at item-level (impression-level in RTB).

3 CTR Distribution Modeling

RTB is generally a two-phase process: 1) CTR estimation based on features of bid request; 2) bid price determination based on estimated CTR. In this section, we explicitly model the CTR distribution in order to deal with the uncertainty of a CTR estimation. Next, in Section 4 we shall propose risk-aware bidding strategies from the inferred CTR distribution and the market price distribution.

In this paper, we don’t provide down-to-earth background of RTB. For more details about RTB, we refer to [33]. For clarity, we summarise notations in this paper in Table 1.

3.1 Preliminary: Bayesian Logistic Regression

We propose to use a Bayesian logistic regression to model the CTR distribution due to the following reasons: (i) logistic regression (LR) has been widely deployed as the CTR prediction model in most RTB ad platforms [17, 25] and our model is a natural extension to tackle the uncertainty of a CTR estimation with LR; (ii) We adopt Bayesian treatment to model uncertainty since it has been well studied by previous works [41, 13] for CTR estimation; (iii) Although Bayesian probit regression [41, 13] has the potential to model the uncertainty, its probit activation function is of no closed form, and thus is computationally low cost effectiveness in RTB.

For readability, in this section, we present a preliminary on Bayesian logistic regression, while for details, we refer to [7]. For a multi-dimensional feature vector $\bm{x}$ representing the input ad display opportunity, the conventional logistic regression estimates the CTR by:

[TABLE]

where $\sigma$ is the sigmoid function and $\bm{w}$ is the weight vector of logistic regression. The likelihood of observing the correct binary click label $y$ given features $\bm{x}$ and weights $\bm{w}$ is

[TABLE]

In the Bayesian version of logistic regression, $\bm{w}$ is modeled as a random variable with a p.d.f. $p(\bm{w})$. Thus the marginal conditional probability $p(y|\bm{x})$ is

[TABLE]

We follow [41, 13] to adopt a Gaussian prior $N(\bm{\mu_{0}},q_{0}^{-1}\bm{I})$ on $\bm{w}$, which is a practical setting. After observing a data instance $(\bm{x},y)$, the posterior distribution of $\bm{w}$ becomes

[TABLE]

where $\mu_{\text{old},i}$ and $q_{\text{old},i}$ are the prior parameters of the $i$-th dimension of $\bm{w}$ before observing the data instance $(\bm{x},y)$.

Approximation of Posterior. Since the posterior Eq. (4) is computationally complex, we maintain a Laplace approximation [7] to keep it consistent with the prior. There are alternative approximate inferences such as variational inference (VI) [7]. In this paper we adopt Laplace approximation due to its simpler implementation and lower computational cost than VI, which involves extra variational parameters and invokes a time-consuming EM algorithm for training. The Gaussian approximation to the posterior distribution takes the form

[TABLE]

where $\bm{\mu}_{\text{new}}$ is defined by the $\bm{w}_{\text{MAP}}$ which maximizes the logarithmic posterior:

[TABLE]

whose SGD updating is

[TABLE]

and $\bm{S}_{\text{new}}$ is given by the inverse of the matrix of second derivatives of the negative log likelihood, which satisfies

[TABLE]

As we follow [41, 13] to assume each $w_{i}$ as independent with each other and thus $\bm{S}$ is diagonal, we have the updating of precision parameters as

[TABLE]

3.2 Predicted CTR Distribution

Equipped with a Bayesian logistic regression with a Laplace approximation of the parameter posterior, we are ready to propose a simple yet novel solution for modeling the CTR distribution. Note that the CTR itself is a probability estimation that a click event occurs from an impression with feature $\bm{x}$ takes the form

[TABLE]

where $\hat{y}$ denotes the CTR random variable that generates the binary observation $y$, so our goal is to estimate the distribution of $\hat{y}$. As introduced in Section 3.1, we assume each $w_{i}$ is from Gaussian i.i.d., the distribution of $\sum_{i}w_{i}x_{i}$ also follows $N(\sum_{i}\mu_{i}x_{i},\sum_{i}q_{i}^{-1}x_{i})$. Thus, we have

[TABLE]

Consider that if random variables $x$ and $y$ satisfy $y=g(x)$ and $g^{-1}$ is monotonic and differentiable, we have [37]

[TABLE]

In our case, $\sigma^{-1}(\hat{y})=\ln\hat{y}-\ln(1-\hat{y})$ is monotonic and differentiable within $(0,1)$, so we obtain the closed-form of $\hat{y}$’s p.d.f.

[TABLE]

which provides an explicit CTR p.d.f. To our best knowledge, the above proposed solution has not been studied in previous literature [41, 13, 36]. To understand the nature of the model, Figure 1 plots the CTR distribution against its parameters $\mu$ and $q$, where $\bm{x}=\bm{1}_{10}$. As observed, the p.d.f. presents a single-peak shape in $[0,1]$ which is similar with the shapes of beta distributions.

It is straightforward to see from Figure 1 that $\mu$ and $q$ jointly determine the peak location and sharpness of the CTR p.d.f. and specifically $\mu$ influences more on the peak location while $q$ influences more on the distribution sharpness. To understand how it models the CTR prediction uncertainty/confidence, from Eq. (8), we see each time a data instance with feature $\bm{x}_{i}$ is observed, the precision $q_{i}$ will be updated with a higher value, which in turn contributes a sharper CTR p.d.f. in Eq. (12). Therefore, for the ad impression with frequent (similar) features, the predicted CTR is of low uncertainty, and vice versa.

4 Risk-aware Bidding Strategies

With our CTR distribution model in Eq. (12), we next investigate the conditional distribution of a utility given a specific bid price for an input bid request. By considering a risk-aware utility as an optimization target, we are ready to derive the corresponding risk-aware bidding strategy. Note that alternative CTR distribution models can also be incorporated in our solution framework.

Specifically, we start from the theoretic derivation of a Bayesian truth-telling bidding strategy and provide an analysis of its risk in Section 4.1. Then we will propose two solutions, discussed in Sections 4.2 and 4.3 respectively.

4.1 Analysis: Bayesian Truth-telling Bidding

The utility $r$ of an ad impression could be defined based on the advertiser’s value $v$ on a specific user action, e.g., click or conversion. For example, if the value $v$ is on each click, given an impression with CTR $\hat{y}$ distributed as in Eq. (12), the utility $r$ and its p.d.f. of this impression are

[TABLE]

Moreover, the cost to win the ad impression comes from the highest bid from other competitors, defined as market price $z$ [5]. The profit of winning this impression is then calculated as the utility $r$ minus the cost $z$, which is set as the optimization target of performance-driven campaigns [39].

In a general setting, considering both the estimated CTR $\hat{y}$ and cost $z$ are stochastic variables: $\hat{y}\sim p_{\hat{y}}(\hat{y})$ and $z\sim p_{z}(z)$, the bid optimization problem is to find the optimal bid price to the auction.

Let us first consider a simple case without considering the uncertainty of our estimation, where the goal is to maximize the expected profit $R(b)$ by marginalizing out $\hat{y}$ and $z$:

[TABLE]

Take the derivative w.r.t. $b$ and set it to 0:

[TABLE]

We see that the optimal bid price is the product of the action value and the estimated CTR $\hat{y}$, which is independent of the market price distribution. If we assume $\hat{y}$ is known and fixed, i.e., $p_{\hat{y}}(\hat{y})$ focuses its mass on a single point, then the optimal bid price is $v\cdot\hat{y}$. Note that the optimality of truth-telling bidding is for one-shot auction. When considering campaign budget and auction volume, a coefficient $\phi$ is commonly added to Eq. (18) as $\phi\cdot\mathbb{E}[r]$.

Discussion of Risk. The above classic bidding solution is built on maximizing the expectation of the profit $R(b)$, regardless of its uncertainty. However, a potential problem is that there is a chance a bid is won, but $v\cdot\hat{y}$ is less than $z$, in which case $R(b)$ will be negative. We can obtain the probability of such negative profit $P(R(b)<0)$ as

[TABLE]

which shows it is of risk to get negative profit whatever the positive bid price. We illustrate our point in Figure 2 using an example. We set the CTR p.d.f. as Eq. (12) with $\sum_{i}\mu_{i}x_{i}=-1$, $\sum_{i}q_{i}^{-1}x_{i}=\frac{1}{3}$; the market price distribution is assumed to be a log-normal p.d.f. with $\mu=4$, $\sigma=0.5$; the value per click is set as 300. The expected value of CTR is 0.283, and thus the truth-telling bidding in Eq. (18) would give 84. From our simulation with 10,000 samples, we plot the profit distribution given the truth-telling bid 84 with probability 16.7% the profit is negative. From Eq. (19) we see that the down-side risk, the probability of the CTR lower than the mean $\hat{y}$, contributes more to the probability of negative profit; the higher standard deviation (or variance) is of $\hat{y}$, the higher probability of such negative profit may occur in the truth-telling bidding case. Thus, such risk of negative profit should be carefully considered and incorporated into the bidding strategies to help campaigns achieve satisfactory performance with controlled risk.

4.2 Bidding with Value-at-Risk Utility

The utility p.d.f. $p_{r}(r)$ is of high importance for risk modeling. From Figure 2 we know that due to the uncertainty of utility, bidding with the utility expectation $\mathbb{E}[r]$ will introduce the risk of negative profit, which might be worse than no bid. Let us examine the downside(upside) risk.

Lemma 1** (Cantelli’s Inequality)**

For a random variable $r$ with mean $\mu$ and standard deviation $\sigma$, the following inequalities hold:

[TABLE]

With Lemma 1 we can find a variant of value at risk (VaR) [19, 27] for utility

[TABLE]

which has a guarantee that for $\alpha>0$ the real utility is lower than $\tilde{r}$ with a probability smaller than $1/(1+\alpha^{2})$; for $\alpha<0$ the real utility is higher than $\tilde{r}$ with a probability smaller than $1/(1+\alpha^{2})$.

Using VaR $\tilde{r}$ as a new risk-aware utility, we set the truth-telling bid at $\tilde{r}$. Taking Eq. (13) into Eq. (22) gives

[TABLE]

where the p.d.f. of $\hat{y}$ is given in Eq. (12).

When $\alpha>0$, the bidding strategy is risk-averse, inversely when $\alpha<0$ it is risk-seeking; the traditional truth-telling bidding (Eq. (18)) is now a special case of $b_{\text{VaR}}(r)$ with $\alpha=0$, called risk-neutral.

Note that we derive VaR strategy based on Lemma 1 rather than the distribution of $\hat{y}$ (Eq. (12)) directly for the following reasons: (i) Lemma 1 is a general one in the sense that it only needs mean and standard deviation, so it can be applied to other CTR distribution models, especially those without analytical forms; (ii) $\alpha$ can be simply set over all bid requests to make the computation more efficient. The uniform $\alpha$ can also be regarded as a risk control parameter.

Considering campaign budget and auction volume, a coefficient $\phi$ is commonly added to Eq. (23), which is similar to truth-telling bidding [25]. This also applies to the strategy proposed in Section 4.3. The detailed realization method and efficiency analysis of both strategies are provided in the appendix.

4.3 Bidding for Risk Management of Profit

Another way to approaching risk-aware bidding strategies is to go back to the analysis of campaign profit $R(b)$, which is one of the key performance indicators (KPIs) in RTB.

Mathematically,

[TABLE]

As both CTR $\hat{y}$ and market price $z$ are modeled as stochastic random variables with p.d.f. $p_{\hat{y}}(\hat{y})$ and $p_{z}(z)$, $R(b)$ can be naturally regarded as a dependent random variable. Again, using Lemma 1, the value-at-risk of profit

[TABLE]

which has a guarantee that for $\alpha>0$ the real profit is lower than $\tilde{R}(b)$ with a probability smaller than $1/(1+\alpha^{2})$; symmetrically for $\alpha<0$.

In this setting, we propose the second risk-aware bidding strategy that generates the bid which yields the maximum value-at-risk of profit:

[TABLE]

Through analyzing the properties of $R(b)$, we find that $\mathbb{E}[R(b)]$ and $\operatorname*{\text{Std}}[R(b)]$ have a non-trivial trade-off relation, which can be balanced by $b$. We solve the optimal $b$ by importing the concept of efficient frontier from finance [16]. Detailed analysis can be found in the appendix.

5 Offline Empirical Study

In this section, we empirically evaluate our proposed risk-aware bidding strategies on real-world data111Our experiment is repeatable and the code is given below: https://github.com/pkuzhf/bidding-at-risk-experiment.

5.1 Datasets

Two ad log datasets in our empirical study are:

iPinYou

is the largest independent programmatic media buying platform in China. The dataset222Dataset link: http://data.computational-advertising.org was released for research after its global competition on RTB algorithms in 2013 [18]. It contains 19.50M ad impressions, 14.89K clicks from 9 campaigns during 10 days in 2013, which involve 16K Chinese Yuan (CNY) expense in total. According to the data publishers, the last three day data of each campaign is split into test data while the former part into training data. The overall effective cost per click (eCPC) is 1.07 CNY on the training data and 1.13 CNY on the test data, which means the overall market competition did not change dramatically during that 10 days.

YOYI

is another mainstream demand-side platform company. This proprietary dataset is mainly used for training the risk-aware models deployed for online A/B testing. Details will be given in Section 6.

Both datasets are with record-per-line format. After ad log joining, each record is formalized as a triple $(\bm{x},y,z)$, where $\bm{x}$ is the high dimensional feature vector for each bid request with the corresponding ad information, $y$ is the user feedback on the ad impression, e.g., the binary click or conversion actions, $z$ is the market price for that auction, i.e., the lowest price to bid in order to win the auction.

5.2 Experiment Protocol

We followed [39] to set up our evaluation procedure. For each campaign bidding strategy with a predefined budget, the optimal parameters $(\bm{\mu},\bm{S})$ were learned on training data and the hyperparameters ($\alpha$ and $\phi$, discussed later) were tuned on validation data, which was the early half split from the test data. Then we replayed the historic bid records to test the performance on the other half of test data. We did not use cross-validation since our data is sequential.

For each campaign, there was a defined value $v$ for the user action, i.e., a click in our experiment. Following [39] the click value is defined by a proportion of eCPC on training data to imitate the true value set by the advertisers. In this paper, we set the proportion to $100\%$. Every time the tested bidding agent received a bid request from the ad exchange, it generated a bid with the tested bidding strategy. If the bid price was higher than the historic market price, the agent won the ad impression and paid the market price and then the historic recorded binary user feedback, i.e., click or not, was observed. If there was a user click, the agent made revenue of the click value. The test ended either when there was no more test bid request or when the campaign budget was exhausted, if applicable.

Note that the offline experiment cannot fully simulate the market competition because there is no observed user feedback and market price for historically lost auctions. Nevertheless, our evaluation protocol keeps the bid requests, displayed ads, and auction environment unchanged. We try to answer that under the same context if the campaign was given a different bidding strategy, whether they would be able to get more clicks with the budget limitation.

5.3 Compared Bidding Strategies

The compared bidding strategies are as follows.

• •

LR - Linear Revenue Bidding: the baseline, which is the most widely used bidding function

[TABLE]

where $\hat{y}$ is the predicted CTR with logistic regression as in Eq. (1). $\phi$ is the scaling parameter tuned based on the market competitiveness [25]. The overall AUC of this LR estimator on iPinYou dataset is $69\%$.

• •

VaR - Value at Risk Bidding: our first risk-aware bidding strategy proposed in Section 4.2, which bids the value at risk, shown in Eq. (23), where $\alpha$ is the hyperparameter to be tuned on validation data.

• •

RMP - Risk Management of Profit based Bidding: the second risk-aware bidding strategy proposed in Section 4.3, which seeks the optimal bid to maximize the expected profit with its standard deviation as risk constraint, shown in Eq. (26), where $\alpha$ is the hyperparameter to be tuned on validation data.

The two risk-aware bidding strategies rely on the same CTR distribution model proposed in Section 3. The initial prior distribution of $\bm{w}$ was given that (i) $\bm{\mu}_{0}$ was set as the point estimation generated by logistic regression; (ii) $q_{0}$ was set as constant 1. Our model achieves a comparable AUC with standard LR model. Note that for the budgeted bid optimization tasks in Section 5.7, there would also be a hyperparameter $\phi$ multiplied in VaR (23) and RMP (26) to be tuned to avoid budget under- or over-delivery.

5.4 Evaluation Measures

For a bid optimization task, the major evaluation measures are campaign profit, i.e., the earned total click value minus the cost, and ROI, i.e., the ratio between the profit and the cost. In addition, we also monitor other key metrics, such as cost per thousand impressions (CPM), auction winning rate, CTR and eCPC to gain more insights.

In order to investigate how bidding strategies balance return against risk, we also propose an additional metric: profit - $\lambda$ cost, which is named as Cost-Penalized Profit (CP-Profit) in this paper. Intuitively, advertisers want to maximize the profit of their performance-driven campaign given the budget, or minimize the advertising cost given the profit, either of which forms a profit/cost tradeoff. Note that this metric was mainly used for model selection, i.e., selecting $\alpha$ and $\phi$. As we will show in Section 5.5 about the profit/cost analysis, CP-Profit is an effective metric to balance the profit and the cost to select an optimal model.

5.5 Profit and Cost Analysis

We analyzed the profit and cost of various bidding strategies with various parameter settings. This would help us understand the properties of the bidding strategies and help us select optimal model hyperparameters.

Non-Budgeted Settings. Without budget constraint, the basic LR bidding strategy is truth-telling, i.e. $\phi=1$. For the risk-aware bidding strategies we varied the value of $\alpha$. Figure 3 shows the profit and cost change when setting different $\alpha$’s in VaR Eq. (23), with $\alpha>0$ as risk-averse, $\alpha<0$ as risk-seeking and $\alpha=0$ as risk-neutral. We clearly observed that for all tested campaigns the results generally formed a single-peak concave shape, showing strong trade-off between the profit and cost. Risk-averse strategies intended to yield lower cost than risk-seeking ones, as the risk-averse strategies would always bid lower than the risk-neutral one, and the latter further bid lower than the risk-seeking ones. The performance-driven campaigns pursue high profit and low cost. Thus if a bidding strategy yields higher profit and lower cost than another one, we say the former strategy dominates the latter one. From the results we saw that almost all the risk-seeking strategies were dominated by part of the risk-averse strategies. That is to say, for almost every risk-seeking setting with $\alpha<0$, there always exists one or more risk-averse setting with $\alpha>0$ that yields equal or higher profit with lower cost.

Figure 4 illustrates the profit and cost trade-off yielded by RMP Eq. (26) with various risk settings. We also observed that, similarly, the risk-averse strategies tended to yield lower cost than the risk-seeking ones. The risk-neutral strategy acted as a splitting point between the risk-averse and risk-seeking ones.

Budgeted Settings. In practice, advertisers tend to set up campaign budget constraints. With the budget constraint, the basic LR bidding strategy is not necessarily truth-telling, i.e., $\phi\neq 1$ in Eq. (27) [25]. We performed an analysis on the profit and cost trade-off for different budget constraints. For each campaign, we followed [39] to use $1/2,1/4,\ldots,1/32$ of the original total cost in the test data as the budget. For LR we varied the value of $\phi$ while for VaR and RMP we varied both $\phi$ and $\alpha$ to obtain different profits and costs. As an example, the results of iPinYou campaign 3427 for RMP with 1/4 budget setting are shown in Figure 5. We saw that part of the risk-averse strategies dominated other strategies, which was in accordance with the ones without budget constraints.

Distinction with Lowering Bidding. The performance of risk-averse bidding strategies was not due to lowering bidding. As we know, for a truth-telling linear bidding, if we lower the bid price uniformly, it would result in cost decreasing, profit decreasing and ROI increasing. Unlike this, our risk-averse bidding strategies lowered the bid price of uncertain requests and raised the bid price of confident requests, which led to higher profit and higher ROI with invariant cost. It can be verified in Figure 5. For example, the left circled LR model lowered the bid price of the right circled LR model by applying a smaller $\phi$, so it achieved lower cost, lower profit and higher ROI. In the meantime, the risk-averse models above the right circled LR model, which had a positive $\alpha$ and a larger $\phi$, achieved the same cost, higher profit and higher ROI.

5.6 Bidding without Budget Constraints

In order to select different risk-aware models (i.e. the strategies’ hyperparameter $\alpha$) with various risk-return balanced metrics, we used the proposed CP-Profit metric. Specifically, with a fixed value of $\lambda$, the contour of a specific CP-Profit is a straight line in the cost-profit plot. The highest CP-Profit is achieved by the tangent point of the contour and the cost-profit points. Figure 6 provides example tangent points with different $\lambda$’s on the cost-profit plot. For a fixed $\lambda$, we then obtained an optimal $\alpha$ that maximizes the CP-Profit.

Figure 7 plots the performance of the tested bidding strategies against CP-Profit metrics with various $\lambda$ in validation data. We found that: (i) The risk-aware strategies VaR and RMP outperformed the baseline LR on all studied campaigns with all $\lambda$ settings. (ii) Specifically, when $\lambda=0$, i.e., the evaluation metric was purely the profit, VaR and RMP outperformed LR with 50.7% and 25.1% improvements. Also note that there were $89\%$ selected models with positive $\alpha$’s, which suggested when there is a fixed volume of bid requests and no limited budget, the risk control helps advertisers make higher profit by spending less money on opportunities with high risk. (iii) For metrics with $\lambda>0$, the performance gained of VaR and RMP over LR were larger. As we observed that the CP-Profit curves of VaR and RMP were convex while the ones of LR was a straight line, which meant the risk-aware strategies provided higher ROI when the model selection metrics became more conservative, i.e., with higher $\lambda$.

With the model selected on validation data, we compared the strategies on test data. The overall performance across 9 iPinYou campaigns is shown in Figure 8. We observed that: (i) Most risk-aware bidding strategies yielded significantly higher profit than the baseline LR, which demonstrated the effectiveness of incorporating risk modeling in bid optimization. (ii) All VaR strategies yielded significantly higher ROI compared to LR, which meant bidding at risk helps advertisers avoid from wasting money on risky opportunities. It turned out that such risky opportunities brought little revenue so avoiding them led to a lower cost and higher profit, and thus much higher ROI. (iii) RMP strategies stably yielded about 10% profit more than LR while the ROI was lower than LR. This was because RMP sought the bid price which directly maximized the value at risk of profit, which resulted in consistent profit gains. On the other hand, RMP did not explicitly control the bid price like VaR, thus it did not yield lower CPM or cost. As a result, its ROI was not higher than LR. (iv) All the risk-aware bidding strategies brought higher CTR than the baseline LR, which might be counterintuitive. Actually, the proposed strategies allocate the budget from the high-uncertainty cases to low-uncertainty ones, which does not mean the average CTR should get lower.

5.7 Bidding with Budget Constraints

Following the previous section, we sought the tangent points which maximized CP-Profit with various $\lambda$’s. The selected models are highlighted with red circles in Figure 5. There might be overlapped red circles as the CP-Profit with different $\lambda$’s still selected the same model.

Figure 9 provides the overall performance over 9 iPinYou campaigns with budget constraint (1/2 budget setting). The baseline is LR with $\lambda=0$, i.e., simply selecting $\phi$ yielding the highest profit on validation dataset. We found that: (i) For small $\lambda$ settings, VaR and RMP were both better than LR on profit, which verified the claim that the risk-aware bidding strategies successfully yield higher profit via controlling the risk. Specifically, risk-aware bidding strategies performed better on yielding profit for the campaigns with poorer LR estimators. (ii) All proposed methods, i.e., LR with non-zero $\lambda$ model selection and VaR/RMP with various $\lambda$’s, yielded higher ROI than the baseline LR with $\lambda=0$. Some strategies (VaR and RMP with $\lambda=0,0.2$) yielded both higher profit and higher ROI. (iii) All risk-aware bidding strategies provided higher CTR (except for the conservative VaR with $\lambda=0.4$), which meant the proposed strategies filtered out the low-value cases, which were always with high uncertainty. (iv) For CPM and winning rate, VaR reduced the CPM by lowering bids on unconfident cases and highering the bids on confident ones (via tuning $\phi$); RMP did not guarantee any low bid as it just sought the bid yielding the highest value of risk-controlled profit.

6 Online Deployment and A/B Test

Online Environment. We have deployed our bidding strategies on YOYI Platform, which is a mainstream DSP in the global RTB market. The total daily bid request volume received by YOYI was about $5$ billion. Among the received bid requests, about 200M (4%) met YOYI’s bid target rules of cost-per-click (CPC) campaigns, for which YOYI would return a valid bid. Each of our tested strategies was allocated with $1.5\%$ bid request volume from $10$ CPC campaigns, i.e. about 3M bid opportunities daily. For training our models, we collected the impression and click data of $7$ consecutive days just before the A/B testing in Jan. 2016, which contained 424M ad impressions, 532K clicks and 47.2K US dollars (USD) expense. The eCPC on the training data is $0.087$ USD. A $2\%$ negative instance down sampling was performed by YOYI system. The advertisers of tested CPC campaigns paid a fixed amount for every user landing action, i.e., valid click. The landing value $v$ was set to be $0.054$ USD, which was the average CPC of all campaigns.

Test Setting. We tested two strategies for $7$ days in Jan. $2016$: (i) The baseline LR strategy. It consists of a logistic regression whose AUC is $88\%$ and a linear bidding function with fixed $\phi=0.56$ set by YOYI operation staff. (ii) The proposed VaR strategy. The parameter $\alpha$ acquired from the training was a positive value that slightly varied daily, which meant the trained bidding strategy risk-averse. The parameter $\phi$ was set to be greater than LR. Thus the two parameters had opposite effects on bid price, which resulted in an average bid price close to that of LR. Strategies were tested with the same bid volume constraint. The bid volume for each strategy was $19.5$M. The whole test live volume contained $39$M auctions resulting in $3.3$M impressions and $7.4$K user landings. VaR spent 12% more budget than LR since $\phi$ was learned from training thus there was no guarantee of equivalent budget spending for A/B test.

Result Discussion. The online results are presented in Figure 10. We show the performance over six metrics. From the results, we found that: (i) Our VaR strategy outperformed baseline LR on ROI by about 5%, which demonstrated the efficacy of our risk-aware bidding strategy. (ii) VaR achieved 17.5% higher profit than LR. Although this was partly due to the more budget spent by VaR, the more important fact was that VaR strategy could help advertiser spend such more budget at a equivalent or even higher ROI within the same bid request volume. Note that advertisers determine their budget usually depending on ROI and ROI is always negative correlative with budget. Thereby, if a strategy allows an additional amount of budget spent on the same ROI, advertisers would like to increase their budget by that amount. (iii) VaR won much fewer auctions than LR. According to our setting, VaR bid higher for confident cases and lower for risky cases. Its low winning rate result indicated that the extra confident cases VaR gained were fewer than the risky cases VaR lost. (iv) As for CTR, VaR achieved 64.4% higher CTR than baseline LR, which demonstrated the high efficiency of risk-aware strategies in finding high quality ad display opportunities. (v) VaR got higher CPM than LR. This indicated that our strategy tended to target at impressions with higher quality at the cost of higher price. Although VaR strategy achieved higher CPM, it also achieved higher CTR and the co-effect was reflected in low eCPC and high ROI, which were beneficial.

7 Conclusion

In this paper, we presented a solution for modeling the uncertainty of CTR estimation in RTB display advertising. With a Bayesian logistic regression CTR estimator, we obtained a closed form of CTR distribution density. On the basis of the distributions of CTR and the market price, two risk-aware bidding strategies are formulated: the first one (VaR) bids the value at risk of the estimated utilities, while the second one (RMP) seeks the optimal bid that maximizes a lower bound of profit with a controlled risk. Our risk-return analysis and offline experiment demonstrated 15.4% profit gain over a linear optimized bidding strategy. To test the applicability of our risk-aware bidding strategies in a live setting, the bidding strategies have been deployed on an operational platform. 17.5% profit gain over the baseline was observed in a 7-day online A/B test.

For future work, we will analyze the market equilibrium if more advertisers adopt the risk-aware bidding strategies. We also plan to further explore applications from the proposed CTR distribution model and the risk-aware bid optimization framework. For an ad placement with multi-frame dynamic creatives, a portfolio optimization [11] can be performed to select the optimal creatives combination to yield the highest profit with a controlled risk. Moreover, our CTR distribution model is potential to be dynamically updated to evaluate users’ interest on an advertised product after repeated ad displays over time.

Acknowledgement This work is financially supported by National Natural Science Foundation of China (61632017).

8 APPENDIX

VaR Realization. Our VaR strategy relies on the CTR distribution model from Section 3. The parameters $\mu_{i}$ and $q_{i}$ are trained offline, while $\sum_{i}\mu_{i}x_{i}$ and $\sum_{i}q_{i}^{-1}x_{i}$ for a given bid request is calculated online, which only takes double time compared to a traditional LR model. We then calculate $\mathbb{E}[\hat{y}]$ and $\operatorname*{\text{Std}}[\hat{y}]$ from $\sum_{i}\mu_{i}x_{i}$ and $\sum_{i}q_{i}^{-1}x_{i}$, which is, however, quite costly. Fortunately, we can move this calculation from online to offline by building up a look-up table, with $(\sum_{i}\mu_{i}x_{i},\sum_{i}q_{i}^{-1}x_{i})$ as its key and $(\mathbb{E}[\hat{y}],\operatorname*{\text{Std}}[\hat{y}])$ as its value. Since $\sum_{i}\mu_{i}x_{i}$ and $\sum_{i}q_{i}^{-1}x_{i}$ are in particular ranges respectively, we can discretize the value with enough accuracy and calculate $\mathbb{E}[\hat{y}]$ and $\operatorname*{\text{Std}}[\hat{y}]$ of the discrete $(\sum_{i}\mu_{i}x_{i},\sum_{i}q_{i}^{-1}x_{i})$ combinations. For each bid request online, we round ($\sum_{i}\mu_{i}x_{i}$, $\sum_{i}q_{i}^{-1}x_{i}$) to get the discrete key and then read $\mathbb{E}[\hat{y}]$ and $\operatorname*{\text{Std}}[\hat{y}]$ from the look-up table with $O(1)$ time.

VaR Efficiency. For offline look-up table building, if we separate $\sum_{i}\mu_{i}x_{i}$ and $\sum_{i}q_{i}^{-1}x_{i}$ into 1000 bins respectively and sample $1000$ times from the corresponding $\hat{y}$ distribution, it needs $10^{9}$ sampling operations in total, which can be done in a few minutes on a modern PC. For online bidding, as discussed above, it has equal computational complexity with logistic regression.

RMP Realization. We solve Eq. (26) using a look-up table too, which takes $(\sum_{i}\mu_{i}x_{i},\sum_{i}q_{i}^{-1}x_{i})$ as its key and bid price $b$ as its value. For building the look-up table, we offline enumerate $b$ within a limited range. For each enumerated $b$, we sample market price $z$ and estimated CTR $\hat{y}$ according to their distributions for sufficient times and calculate $R(b)$ for each sample. After that we obtain the objective $\mathbb{E}[R(b)]-\alpha\operatorname*{\text{Std}}[R(b)]$ for each $b$. Finally we choose the optimal $b$ that maximizes the objective and store the solution in the look-up table. When online bidding, we just calculate $\sum_{i}\mu_{i}x_{i}$ and $\sum_{i}q_{i}^{-1}x_{i}$ for the bid request, then find the optimal $b$ from the look-up table with $O(1)$ time.

RMP Efficiency. Compared to VaR strategy, RMP additionally enumerates $b$ and samples $z$ in offline calculation. In practice, $1000$ enumerations are enough as the range of bid price is quite small. Note that, for each key of the look-up table, all the enumerated bid prices $b$ can share the same samples of $\hat{y}$ and $z$. Thus, the computational complexity of offline calculation for RMP only has constant difference with the one for VaR, which takes a few minutes as discussed above. When online bidding, RMP strategy has equal computational complexity with logistic regression, which makes it capable in real-world business.

Analysis of RMP strategy. To give an insight into solving $b$ in Eq. (26), we analyze the relationship between $\mathbb{E}[R(b)]$ and $\operatorname*{\text{Std}}[R(b)]$. Given a bid $b$, the expectation of $R(b)$ is formulated as

[TABLE]

where $z_{k}$ denotes $\int_{0}^{b}z^{k}p_{z}(z)dz$ for simplicity. And the expected squared profit is

[TABLE]

Based on $\mathbb{E}[R(b)]$ and $\mathbb{E}[R(b)^{2}]$, the variance of profit is

[TABLE]

We further analyze the change trends of the expectation $\mathbb{E}[R(b)]$ and variance $\operatorname*{\text{Var}}[R(b)]$ w.r.t. the bid price $b$.

[TABLE]

which is positive when $b<v\mathbb{E}[\hat{y}]$ and negative otherwise.

We are also interested in the change trend of $\operatorname*{\text{Var}}[R(b)]$ in the range around the truth-telling bid price. We obtain

[TABLE]

Therefore, the variance increases in a range around the conventional truth-telling bid price $v\mathbb{E}[\hat{y}]$, so does the standard deviation. Figure 11 shows an example of the relationship between $\mathbb{E}[R(b)]$ and $\operatorname*{\text{Std}}[R(b)]$, where the parameters of CTR distribution and market price distribution are the same as Figure 2.

Based on the reasonable preference of higher $\mathbb{E}[R(b)]$ and lower $\operatorname*{\text{Std}}[R(b)]$, we further borrow the concept of efficient frontier from finance [16]. Every point at the efficient frontier corresponds to the optimal bid price defined by Eq. (26) with a particular $\alpha$. For example, the slopes $\alpha$ of the dashed lines are $0,1,2$ respectively. The tangent points are of the maximum $\mathbb{E}[R(b)]-\alpha\operatorname*{\text{Std}}[R(b)]$, each of which corresponds to a particular bid price, i.e., the solution of Eq. (26).