Sequential Counterfactual Risk Minimization

TL;DR

This paper extends Counterfactual Risk Minimization to a sequential setting where policies can be deployed multiple times, introducing a new estimator and demonstrating improved theoretical and empirical performance.

Contribution

It proposes Sequential Counterfactual Risk Minimization (SCRM), extending CRM theory to multiple deployments and introducing a novel estimator for better performance.

Findings

Improved excess risk and regret rates with multiple deployments

Empirical validation in discrete and continuous action spaces

Demonstrated benefits of multiple policy deployments

Abstract

Counterfactual Risk Minimization (CRM) is a framework for dealing with the logged bandit feedback problem, where the goal is to improve a logging policy using offline data. In this paper, we explore the case where it is possible to deploy learned policies multiple times and acquire new data. We extend the CRM principle and its theory to this scenario, which we call "Sequential Counterfactual Risk Minimization (SCRM)." We introduce a novel counterfactual estimator and identify conditions that can improve the performance of CRM in terms of excess risk and regret rates, by using an analysis similar to restart strategies in accelerated optimization methods. We also provide an empirical evaluation of our method in both discrete and continuous action settings, and demonstrate the benefits of multiple deployments of CRM.

Tables

Table 1. Table 1: Needed sample size to achieve test loss L ​ ( θ ) ≤ p ∗ L ​ ( θ ∗ ) 𝐿 𝜃 𝑝 𝐿 superscript 𝜃 L(\theta)~{}\leq~{}p*L(\theta^{*}) on the setting in Example 3.1 over the average of 10 random runs. SCRM needs way less data to converge to near optimal solution. λ 𝜆 \lambda is set to its theoretical value.

Percentage $p$	$0.7$	$0.8$	$0.9$
CRM	$100\times 2^{10}$	$100\times 2^{16}$	$>100\times 2^{22}$
SCRM (ours)	$100\times 2^8$	$100\times 2^9$	$100\times 2^{11}$

Table 2. Table 2: Test loss after 10 rollouts when choosing λ 𝜆 \lambda by a posteriori selection ( λ ′ superscript 𝜆 ′ \lambda^{\prime} ) or with proposed heuristic ( λ ^ ^ 𝜆 \hat{\lambda} ). Our heuristic is competitive with the a posteriori selection of a fixed λ ′ superscript 𝜆 ′ \lambda^{\prime} .

	Pricing	Advertising	Yeast	TMC2007
${\lambda }^{\prime }$	$-5.353\pm .178$	$-.716\pm .020$	$.294\pm .026$	$.146\pm .012$
$\hat{\lambda }$	$-5.575\pm .036$	$-.726\pm .001$	$.299\pm .039$	$.164\pm .021$

Table 3. Table 3: Test loss ± plus-or-minus \pm stddev of different methods after 10 rollouts. SCRM achieves optimal or near optimal performance in all datasets. Batch bandit methods did not finish (DNF) on large scale settings, and RL methods perform overall poorly on discrete settings with large action space.

	Pricing	Advertising	Scene	Yeast	TMC2007
$n/\left|𝒜\right|/dim\left(𝒳\right)|$	${10}^5/\infty /10$	${10}^5/\infty /2$	${2.10}^3/2^6/295$	${2.10}^3/2^{14}/104$	${3.10}^4/2^{22}/{3.10}^4$
Baseline	$-3.414\pm .162$	$-.431\pm .120$	$.353\pm .009$	$.478\pm .014$	$.511\pm .003$
SBPE	DNF	DNF	$\text{.179}\pm .001$	$.302\pm .003$	DNF
BKUCB	DNF	DNF	$.236\pm .014$	$.303\pm .004$	DNF
TRPO	$\text{-5.750}\pm .020$	$-.670\pm .030$	$.376\pm .001$	$.434\pm .001$	$.396\pm .001$
PPO	$-5.274\pm .200$	$-.637\pm .015$	$.206\pm .001$	$.463\pm .001$	$.263\pm .001$
CRM	$-5.325\pm .068$	$-.594\pm .100$	$.233\pm .031$	$.362\pm .044$	$.158\pm .034$
SCRM (ours)	$-5.575\pm .036$	$\text{-.726}\pm .020$	$.219\pm .009$	$\text{.294}\pm .026$	$\text{.146}\pm .012$
Skyline	$-5.830\pm .020$	$-.739\pm .002$	$.179\pm .002$	$.312\pm .003$	$.142\pm .001$

Equations

$$L(\theta )=E_{x,\theta ,y}\left[y\right].$$

$$L(\theta )=E_{x,{\theta }_0,y}\left[y\frac{{\pi }_{\theta }(a|x)}{{\pi }_{{\theta }_0}(a|x)}\right],$$

$$\hat{\theta }\in \underset{\theta \in \Theta }{\mathrm{argmin}}{\mathcal{L}}_0(\theta ),$$

$${\hat{L}}_0^{\text{IS}}(\theta )=\frac{1}{n}\sum _{i=1}^n{y}_i\min \left(\frac{{\pi }_{\theta }(a_i|x_i)}{{\pi }_i},\alpha \right),$$

$${\hat{V}}_0^{\text{IS}}(\theta )=\frac{1}{n-1}\sum _{i=1}^n({\chi }_i(\theta )-\bar{\chi }(\theta ){)}^2.$$

$${\Delta }_m=L({\theta }_m)-L({\theta }^{\ast }),$$

$$R_n=\sum _{m=0}^M{\Delta }_m{n}_{m+1}.$$

$${\hat{\theta }}_{m+1}\in \underset{\theta \in \Theta }{\mathrm{argmin}}{\mathcal{L}}_m(\theta ),$$

$${\hat{L}}_m^{\text{IPS-IX}}(\theta )=\frac{1}{n_m}\sum _{i=1}^{n_m}\frac{{\pi }_{\theta ,i}}{{\pi }_{m,i}+\alpha {\pi }_{\theta ,i}}{y}_{m,i},$$

$${\hat{V}}_m^{\text{IPS-IX}}(\theta )=\frac{1}{n_m-1}\sum _{i=1}^{n_m}({\zeta }_i(\theta )-\bar{\zeta }(\theta ){)}^2.$$

$$L(\theta )\le {\hat{L}}_m^{\text{IPS-IX}}(\theta )+{\lambda }_m\sqrt{\frac{{\hat{V}}_m^{\text{IPS-IX}}(\theta )}{n_m}}+\frac{2{\lambda }_m^{2}W}{n_m}+{\delta }_m,$$

$${\mathcal{L}}_m(\theta )={\hat{L}}_m^{\text{IPS-IX}}(\theta )+{\lambda }_m\sqrt{\frac{{\hat{V}}_m^{\text{IPS-IX}}(\theta )}{n_m}},$$

$${\Delta }_{m+1}=L({\theta }_{m+1})-L({\theta }^{\ast })$$

$$\lesssim \sqrt{{\nu }_m^2\frac{d\log n_m-\log \delta }{n_m}}+\frac{W^2+W(d\log n_m-\log \delta )}{n_m},$$

$$\gamma {\mathrm{Var}}_{x,\theta }\left(\frac{{\pi }_{{\theta }^{\ast }}(x|a)}{{\pi }_{\theta }(x|a)}\right)\le {\left(L(\theta )-L({\theta }^{\ast })\right)}^{\beta }.$$

$$\gamma d(\theta ,S_{\Theta }^{\ast })\le {\left(L(\theta )-L({\theta }^{\ast })\right)}^{\beta }$$

$$\Delta_{M}=L(\theta_{M})-L(\theta^{*})\leq O\Big{(}n^{-\frac{1}{2-\beta}}\log n\Big{)}.$$

$$R_{n}=\sum_{m=0}^{M}\Delta_{m}n_{m+1}\leq O\Big{(}n^{\frac{1-\beta}{2-\beta}}\log(n)^{2}\Big{)}.$$

$${\theta }_t\perp \perp {\theta }_{t^{\prime }}|{\theta }_{t-1}.$$

$${\hat{L}}_m^{\text{MIS}}(\theta )=\sum _{t=0}^m\frac{1}{n_t}\sum _{i=1}^{n_t}{\omega }_{t,m}(a_{t,i})y_{t,i}{w}_{t,i}^{\theta },w_{t,i}^{\theta }=\frac{{\pi }_{\theta }(a_{t,i}|x_{t,i})}{{\pi }_{t,i}}.$$

$$E[{\hat{L}}_m^{\text{MIS}}(\theta )]$$

$$=\sum _{t=0}^m{E}_{x,{\theta }_m,y}\left[{\omega }_t(a)y{w}_t^{\theta }\right]$$

$$=\sum _{t=0}^m{E}_{s_1\dots s_{t-1}}\left[E_{x,{\theta }_m,y}\left[{\omega }_t(a)y{w}_t^{\theta }|s_1\dots s_{t-1}\right]\right]$$

$$=\sum _{t=0}^m{E}_{s_1\dots s_{t-1}}\left[E_{x,\theta ,y}\left[{\omega }_t(a)y|s_1\dots s_{t-1}\right]\right]$$

$$=\sum _{t=0}^m{E}_{x,\theta ,y}\left[{\omega }_t(a)y\right]$$

$$=E_{x,\theta ,y}\left[\left(\sum _{t=0}^m{\omega }_t(a)\right)y\right]$$

$$=E_{x,\theta ,y}\left[y\right]$$

$$=L(\theta ),$$

$${\omega }_t(a)=\frac{n_t}{{\sum }_{l=1}^m{n}_l},$$

$${\hat{L}}_m^{\text{n-MIS}}(\theta )=\frac{1}{n}\sum _{t=0}^m\sum _{i=1}^{n_t}{y}_{t,i}\frac{{\pi }_{\theta }(a_{t,i}|x_{t,i})}{{\pi }_{{\theta }_t}(a_{t,i}|x_{t,i})},$$

$$\mathrm{Var}\left[\frac{1}{n}\sum _{t=0}^m\sum _{i=1}^{n_m}{r}_i^m\right]$$

$$=\frac{1}{n^2}\sum _{t=0}^m\mathrm{Var}\left[\sum _{i=1}^{n_m}{r}^m\right]+2\frac{1}{n^2}\sum _{1\le p<q\le m}\sum _{i=1}^{n_p}\sum _{j=1}^{n_q}\mathrm{Cov}\left[r^p,r^q\right]$$

$$=\frac{1}{n^2}\left[\sum _{t=0}^m\mathrm{Var}\left[\sum _{i=1}^{n_m}{r}^m\right]+2\sum _{1\le p<q\le m}{n}_p{n}_q\mathrm{Cov}\left[r^p,r^q\right]\right],$$

$$\hat{V}^{\text{n-MIPS}}_{m}(\theta)=\frac{1}{n^{2}}\left[\sum_{t=0}^{m}\hat{V}(r^{t})+2\sum_{1\leq p<q\leq m}n_{p}n_{q}\left(\frac{1}{n_{p}}\sum_{k=1}^{n_{p}}\big{(}r^{p}_{k}-\bar{r}_{p}\big{)}\big{(}r^{q}_{k}-\bar{r}_{q}\big{)}\right)\right],$$

$${\omega }_t^{BH}(a)=\frac{n_t{\pi }_{{\theta }_t}(a)}{{\sum }_{l=1}^m{n}_l{\pi }_{{\theta }_l}(a)}.$$

Full text

Machine Learning, ICML

1 Introduction

Counterfactual reasoning in the logged bandit problem has become a common task for practitioners in a wide range of applications such as recommender systems (Swaminathan & Joachims, 2015a), ad placements (Bottou et al., 2013) or precision medicine (Kallus & Zhou, 2018). Such a task typically consists in learning an optimal decision policy from logged contextual features and partial feedbacks induced by predictions from a logging policy. To do so, the logged data is originally obtained from a randomized data collection experiment. However, the success of counterfactual risk minimization is highly dependent on the quality of the logging policy and its ability to sample meaningful actions.

Counterfactual reasoning can be challenging due to large variance issues associated with counterfactual estimators (Swaminathan & Joachims, 2015b). Additionally, as pointed out by Bottou et al. (2013), confidence intervals obtained from counterfactual estimates may not be sufficiently accurate to select a final policy from offline data (Dai et al., 2020). This can occur when the logging policy does not sufficiently explore the action space. To address this, one option is to simply collect additional data from the same logging system to increase the sample size. However, it may be more efficient to use already collected data to design a better data collection experiment through a sequential design approach (Bottou et al., 2013, see Section 6.4). It is thus appealing to consider successive policy deployments when possible.

We tackle this sequential design problem and are interested in multiple deployments of the CRM setup of Swaminathan & Joachims (2015a), which we call sequential counterfactual risk minimization (SCRM). SCRM performs a sequence of data collection experiments by determining at each round a policy using data samples collected during previous experiments. The obtained policy is then deployed for the next round to collect additional samples. Such a sequential decision making system thus entails designing an adaptive learning strategy that minimizes the excess risk and expected regret of the learner. In contrast to the conservative learning strategy in CRM, the exploration induced by sequential deployments of enhanced logging policies should allow for improved excess risk and regret guarantees. Yet, obtaining such guarantees is nontrivial and we address it in this work.

In order to accomplish this, we first propose a new counterfactual estimator that controls the variance and analyze its convergence guarantees. Specifically, we obtain an improved dependence on the variance of importance weights between the optimal and logging policy. Second, leveraging this estimator and a weak assumption on the concentration of this variance term, we show how the error bound sequentially concentrates through CRM rollouts. This allows us to improve the excess risk bounds convergence rate as well as the regret rate. Our analysis employs methods similar to restart strategies in acceleration methods (Nesterov, 2012) and optimization for strongly convex functions (Boyd & Vandenberghe, 2004). We also conduct numerical experiments to demonstrate the effectiveness of our method in both discrete and continuous action settings, and how it improves upon CRM and other existing methods in the literature.

2 Related Work

Counterfactual learning from logged feedback (Bottou et al., 2013) uses only past interactions to learn a policy without interacting with the environment. Counterfactual risk minimization methods (Swaminathan & Joachims, 2015a, b) propose learning formulations using a variance penalization as in (Maurer & Pontil, 2009) to find policies with minimal variance. Even so, counterfactual methods remain prone to large variance issues (Dudík et al., 2014). These problems may arise when the logging policy under-explores the action space, making it difficult to use importance sampling tehcniques (Owen, 2013) that are key to counterfactual reasoning. While one could collect additional data to counter this problem, our method focuses on sequential deployments (Bottou et al., 2013, see Section 6.4) to collect data obtained from adaptive policies to explore the action space. Note also that the original motivation is related but different from the support deficiency problem (Sachdeva et al., 2020) where the support of the logging policy does not cover the support of the optimal policy.

Another related literature to our framework is batch bandit methods. Originally introduced by Perchet et al. (2015) and then extended by Gao et al. (2019) in the multi-arm setting, batch bandit agent take decisions and only observe feedback in batches. This therefore differs from the classic bandit setting (Auer et al., 2002; Audibert et al., 2007) where rewards are observed after each action taken by an agent. Extensions to the contextual case have been proposed by Han et al. and could easily be kernelized (Valko et al., 2013). The sequential counterfactual risk minimization problem is thus closely related to this setting. However, major differences can be noted. First, SCRM does not leverage any problem structure as in stochastic contextual bandits (Li et al., 2010) by assuming a linear reward function (Chu et al., 2011; Goldenshluger & Zeevi, 2013; Han et al., ) nor uses regression oracles as (Foster & Rakhlin, 2020; Simchi-Levi & Xu, 2020). Second, deterministic decision rules taken by bandit agents (Lattimore & Szepesvari, 2019) do not allow for counterfactual reasoning or causal inference (Peters et al., 2017), unlike our framework which performs sequential randomized data collection. Third, unlike gradient based methods used in counterfactual methods with parametric policies, batch bandit methods use zero-order methods to learn from data and necessitate approximations to be scalable (Calandriello et al., 2020; Zenati et al., 2022).

The sequential designs that we use are adaptive data collection experiments, which have been studied by Bakshy et al. (2018); Kasy & Sautmann (2021). Closely related to our method is policy learning from adaptive data that has been studied by Zhan et al. (2021) and Bibaut et al. (2021) in the online setting. In contrast, we consider a batch setting and our analysis achieve fast rates in more general conditions. Zhan et al. (2021) use a doubly robust estimator and provide regret guarantees but assume a deterministic lower bound on the propensity score to control the variance. Instead, our novel counterfactual estimator does not require such an assumption. Bibaut et al. (2021) propose a novel maximal inequality and derive thereof fast rate regret guarantees under an additional margin condition that can only hold for finite action sets. Our work instead uses a different assumption on the expected risk, which is similar to Hölderian error bounds in acceleration methods (d’Aspremont et al., 2021) that are known to be satisfied for a broad class of subanalytic functions (Bolte et al., 2007).

In the reinforcement learning literature (Sutton & Barto, 1998), off-policy methods (Harutyunyan et al., 2016; Munos et al., 2016) evaluate and learn a policy using actions sampled from a behavior (logging) policy, which is therefore closely related to our setting. Among methods that have shown to be empirically successful are the PPO (Schulman et al., 2017) and TRPO (Schulman et al., 2015) algorithms which learn policies using a Kullback-Leibler distributional constraint to ensure robust learning, which can be compared to our learning strategy that improves the logging policy at each round. However reinforcement learning models transitions in the states (contexts) induced by the agent’s actions while bandit problems like ours assume that actions do not influence the context distribution. This enables to design algorithms that exploit the problem structure, have theoretical guarantees and can achieve better performance in practice.

Finally, our method is related to acceleration methods (d’Aspremont et al., 2021) where current iterates are used as new initial points in the optimization of strongly convex functions (Boyd & Vandenberghe, 2004). While different schemes use fixed (Powell, 1977) or adaptive (Nocedal & Wright, 2006; Becker et al., 2011; Nesterov, 2012; Bolte et al., 2007; Gaillard & Wintenberger, 2018) strategies, our method differs in that it does not consider the same original setting, does not require the same assumptions nor provides the same guarantees. Eventually, while current models are also used as new starting points, additional data is effectively collected in our setting unlike those previous works that do not assume partial feedbacks as in our case.

3 Sequential Counterfactual Risk Minimization

In this section, we introduce the (CRM) framework and motivate the use of sequential designs for (SCRM).

Notations

For random variables $x\sim\mathcal{P}_{\mathcal{X}}$, $a\sim\pi_{\theta}(\cdot|x)$ and $y\sim\mathcal{P}_{\mathcal{Y}}(\cdot|x,a)$, we write the expectation $\operatorname{\mathbb{E}}_{x,\theta,y}[\cdot]~{}=~{}\operatorname{\mathbb{E}}_{x\sim\mathcal{P}_{\mathcal{X}},a\sim\pi_{\theta}(\cdot|x),y\sim\mathcal{P}_{\mathcal{Y}}(\cdot|x,a)}[\cdot]$ and do the same for the variance $\mathrm{Var}_{x,\theta,y}$. Moreover, $\lesssim$ denotes approximate inequalities up to universal multiplicative terms.

3.1 Background

In the counterfactual risk minimization (CRM) problem, we are given $n$ logged observations $(x_{i},a_{i},y_{i})_{i=1,\ldots,n}$ where contexts $x_{i}\in\mathcal{X}$ are sampled from a stochastic environment distribution $x_{i}\sim\mathcal{P}_{\mathcal{X}}$, actions $a_{i}\sim\pi_{\theta_{0}}(\cdot|x_{i})$ are drawn from a logging policy $\pi_{\theta_{0}}$ with a model $\theta_{0}$ in a parameter space $\Theta$. The losses are drawn from a conditional distribution $y_{i}\sim\mathcal{P}_{\mathcal{Y}}(\cdot|x_{i},a_{i})$. We note $\pi_{0,i}=\pi_{\theta_{0}}(a_{i}|x_{i})$ the associated propensities and assume them to be known. We will assume that the policies in $\pi_{\theta},\theta\in\Theta$ admit densities so that the propensities will denote the density function of the logging policy on the actions given the contexts. The expected risk of a model $\theta$ is defined as:

[TABLE]

Counterfactual reasoning uses the logged data sampled from the logging policy associated to $\theta_{0}$ to estimate the risk of any model $\theta\in\Theta$ with importance sampling:

[TABLE]

under the common support assumption (the support of $\pi_{\theta}$ support is included in the support of $\pi_{\theta_{0}}$). The goal in CRM is to find a model $\theta\in\Theta$ with minimal risk by minimizing

[TABLE]

where $\mathcal{L}_{0}(\theta)=\hat{L}_{0}(\theta)+\lambda\sqrt{\frac{\hat{V}_{0}(\theta)}{n}}$ uses the sample variance penalization principle (Maurer & Pontil, 2009) on samples from $\theta_{0}$ with counterfactual estimates of the expected risk $\hat{L}_{0}$, an empirical variance $\hat{V}_{0}$ and $\lambda>0$. Specifically, in the (CRM) framework, multiple estimators are derived from the IPS method (Horvitz & Thompson, 1952) that uses the following clipped importance sampling estimator of the risk of a model $\theta$ by Bottou et al. (2013); Swaminathan & Joachims (2015a):

[TABLE]

where $\alpha$ is a clipping parameter. Writing $\chi_{i}(\theta)~{}=~{}y_{i}~{}\min(\frac{\pi_{\theta}(a_{i}|x_{i})}{\pi_{0,i}},\alpha)$ and $\bar{\chi}(\theta)=\frac{1}{n}\sum_{i=1}^{n}\chi_{i}(\theta)$ the empirical variance estimator becomes:

[TABLE]

Other estimators aim at controlling the variance of the estimator with self-normalized estimators (Swaminathan & Joachims, 2015b) or with direct methods (Dudik et al., 2011; Dudík et al., 2014) in doubly robust estimators. Even so, the performance of counterfactual learning is harmed when the logging policy under-explores the action space (Owen, 2013). Likewise, counterfactual estimates obtained from a first round of randomized data collection may not suffice (Bottou et al., 2013) to select a model $\hat{\theta}$. In those cases, it could be natural to consider collecting additional samples. While it is possible to use the same logging model $\theta_{0}$ to do so, we will present a framework for designing an improved sequential data collection strategy, following the intuition of sequential designs of Bottou et al. (2013).

3.2 Sequential Designs

In this section we present a design of data collections that sequentially learn a policy from logged data in order to deploy it and learn from the newly collected data. Specifically, we assume that at a round $m\in\{1,\dots M\}$, a model $\theta_{m}\in\Theta$ is deployed and a set $s_{m}$ of $n_{m}$ observations $s_{m}~{}=~{}(x_{m,i},a_{m,i},y_{m,i},\pi_{m,i})_{i=1,\ldots,n_{m}}$ is collected thereof, with propensities $\pi_{m,i}~{}=~{}\pi_{\theta_{m}}(a_{m,i}|x_{m,i})$ to learn a new model $\theta_{m+1}$ and reiterate. In this work, we assume that the loss $y$ is bounded in $[-1,0]$ as in (Swaminathan & Joachims, 2015a) (note however that this assumption could be relaxed to bounded losses) and follows a fixed distribution $\mathcal{P}_{\mathcal{Y}}$. Next, we will introduce useful definitions.

Definition 3.1** (Excess Risk and Expected Regret).**

Given an optimal model $\theta^{*}~{}\in~{}\operatorname*{arg\,min}_{\theta\in\Theta}~{}L(\theta)$, we write for each rollout $m$ the excess risk:

[TABLE]

and define the expected regret as:

[TABLE]

The objective is now to find a sequence of models $\{\theta_{m}\}_{m=1\dots M}$ that have an excess risk and an expected regret $R_{n}$ that improve upon CRM guarantees. To do so, we define a sequence of minimization problems for $m\in\{1,\dots M\}$:

[TABLE]

where $\mathcal{L}_{m}$ is an objective function that we define in Section 4.2. Note that in the setting we consider, samples are i.i.d inside a rollout $m$ but dependencies exist between different sets of observations. From a causal inference perspective (Peters et al., 2017), this does not incur an additional bias because of the successive conditioning on past observations. We provide detailed explanations in Appendix A.1 on this matter. Note also that the main intuition and motivation of our work is to shed light on how learning intermediate models $\theta_{m}$ to adaptively collect data can improve upon sampling from the same logging system by using the same total sample size $n=\sum_{i=0}^{m}n_{m}$. To illustrate the learning benefits of SCRM we now provide a simple example.

Example 3.1** (Gaussian policies with quadratic loss).**

Let us consider Gaussian parametrized policies $\pi_{\theta}=\mathcal{N}(\theta,\sigma^{2})$ and a loss $l_{t}(a)=(a-y_{t})^{2}-1$ where $y_{t}\sim\mathcal{N}(\theta^{*},\sigma^{2})$. We illustrate in Figure 1 the evolution of the losses of learned models $\theta_{m}$ through 15 rollouts with either i) Batch CRM learning on aggregation of data, being generated by the unique initial logging policy $\theta_{0}$ or ii) Sequential CRM learning with models $\theta_{0},\dots,\theta_{m-1}$ deployed adaptively, with data being generated by the last learned model $\theta_{m-1}$ for the batch $m$. We see that the models learned with SCRM take larger optimization steps than the ones with CRM.

We summarize our (SCRM) framework in Algorithm 1 with the different blocks exposed previously. We provide an additional graphical illustration of SCRM compared to CRM in Appendix A.1. In the next section we will define counterfactual estimators from the observations $s_{m}$ at each round and define a learning strategy $\mathcal{L}_{m}$.

4 Variance-Dependent Convergence Guarantees

In this part we aim at providing convergence guarantees of counterfactual learning. We show how we can obtain a dependency of the excess risk on the variance of importance weights between the logging model and the optimal model.

4.1 Implicit exploration and controlled variance

We first introduce a new counterfactual estimator. For this, we will require a common support assumption as in importance sampling methods (Owen, 2013). We will assume that the policies $\pi_{\theta}$ for $\theta\in\Theta$ have all the same support. We then consider the following estimator of the risk of a model $\theta$:

[TABLE]

where $\pi_{\theta,i}=\pi_{\theta}(a_{m,i}|x_{m,i})$ and $\alpha$ is like a clipping parameter which ensures that the modified propensities $\pi_{m,i}+\alpha\pi_{\theta}(a_{m,i}|x_{m,i})$ are lower bounded. Noting \zeta_{i}(\theta)~{}=~{}\big{(}\frac{\pi_{\theta,i}}{\pi_{m,i}+\alpha\pi_{\theta,i}}-1\big{)}y_{m,i}, $\bar{\zeta}(\theta)~{}=~{}\frac{1}{n_{m}}\sum_{i=1}^{n_{m}}\zeta_{i}(\theta)$ we can write the empirical variance estimator as:

[TABLE]

Here, the empirical variance uses a control variate since it uses the expression of $\zeta_{i}(\theta)$ above instead of $y_{m,i}\frac{\pi_{\theta,i}}{\pi_{m,i}+\alpha\pi_{\theta,i}}$. This allows to improve the dependency on the variance in the excess risk provided in Proposition 4.2. Note also that our estimator resembles the implicit exploration estimator in the EXP3-IX algorithm (Lattimore & Szepesvari, 2019), as our motivation is to improve the control of the variance.

4.2 Learning strategy

Next, we aim in this part to provide a learning objective strategy $\mathcal{L}_{m}$, as referred to in Eq. (8). Our approach, like the (CRM) framework, uses the sample variance penalization principle (Maurer & Pontil, 2009) to learn models that have low expected risk with high probability. To do so, we first provide an assumption to be used in our generalization error bound.

Assumption 4.1** (Bounded importance weights).**

For any models $\theta,\theta^{\prime}\in\Theta$ and any $(x,a)\in\mathcal{X}\times\mathcal{A}$, we assume ${\pi_{\theta}(a|x)}/{\pi_{\theta^{\prime}}(a|x)}\leq W$, for some $W>0$.

This assumption has been made in previous works (Kallus & Zhou, 2018; Zenati et al., 2020) and is reasonable when we consider a bounded parameter space $\Theta$. Next, we state an error bound for our estimator.

Proposition 4.1** (Generalization Error Bound).**

Let $\hat{L}^{\text{IPS-IX}}_{m}$ and $\hat{V}^{\text{IPS-IX}}_{m}$ be the empirical estimators defined respectively in Eq. (9) and Eq. (10). Let $\theta\in\Theta$, $\delta\in(0,1)$, and $n_{m}\geq 2$. Then, under Ass. 4.1, for $\lambda_{m}=\sqrt{18(C_{m}(\Theta)+\log(2/\delta))}$, with probability at least $1-\delta$:

[TABLE]

where $C_{m}(\Theta)$ is a metric entropy complexity measure defined in App. B.1 and $\delta_{m}=\sqrt{\log(2/\delta)/({2n_{m}})}$.

This Proposition is proved in Appendix B.2 and essentially uses empirical bounds (Maurer & Pontil, 2009). By minimizing the latter high-probability upper bound, we can find models $\theta$ with guarantees of minimizing the expected risk. Therefore, at each round, we minimize the following loss:

[TABLE]

where $\lambda_{m}>0$ is a positive parameter. Unlike deterministic decision rules used for example in UCB-based algorithms (Lattimore & Szepesvari, 2019), the exploration is naturally guaranteed by the stochasticity of the policies we use.

4.3 Excess risk upper bound

Eventually, we establish an upper bound on the excess risk of the IPS-IX estimator for counterfactual risk minimization using the learning strategy that we just defined. For this, we require an assumption on the complexity measure.

Assumption 4.2**.**

We assume that the set $\Theta$ is compact and that there exists $d>0$ such that $C_{m}(\Theta)\leq d\log(n_{m}).$

This assumption states that the complexity grows logarithmically with the sample size. It holds for parametric policies so long as the propensities are lower bounded, which is verified using our estimator. We now state our variance-dependent excess risk bound.

Proposition 4.2** (Excess Risk Bound).**

Let $n_{m}\geq 1$ and $\theta_{m}\in\Theta$. Let $s_{m}$ be a set of $n_{m}$ samples collected with policy $\pi_{\theta_{m}}$. Then, under Assumptions 4.1 and 4.2, a minimizer $\theta_{m+1}$ of Eq. (11) on the samples $s_{m}$ satisfies the excess risk upper-bound: w.p. $1-\delta$

[TABLE]

where $\nu_{m}^{2}=\mathrm{Var}_{x,\theta_{m}}\left(\dfrac{\pi_{\theta^{*}}(a|x)}{\pi_{\theta_{m}}(a|x)}\right)$.

The proof is postponed to Appendix B.2. The modified propensities in IPS-IX as well as the control variate used in the variance estimator allow us to improve the dependency in $\nu_{m}^{2}$, compared to $\nu_{m}^{2}+1$ obtained in previous work (Zenati et al., 2020). This turns out to be a crucial point to use these error bounds sequentially as in acceleration methods since $\nu_{m}\to 0$ if $\theta_{m}\to\theta^{*}$, as explained in the next section.

5 SCRM Analysis

In this section we provide the main theoretical result of this work on the excess risk and regret analysis of SCRM. We start by stating an assumption that is common in acceleration methods (d’Aspremont et al., 2021) with restart strategies (Becker et al., 2011; Nesterov, 2012) that we will require to achieve the benefits of sequential designs.

Assumption 5.1** (Hölderian Error Bound).**

We assume that there exist $\gamma>0$ and $\beta>0$ such that for any $\theta\in\Theta$, there exists $\theta^{*}\in\operatorname*{arg\,min}_{\theta\in\Theta}L(\theta)$ such that

[TABLE]

Typically, in acceleration methods, Hölderian error bounds (Bolte et al., 2007) are of the form:

[TABLE]

for some $\gamma,\beta>0$ and where $d(\theta,S_{\Theta}^{*})$ is some distance to the optimal set ($S_{\Theta}^{*}=\operatorname*{arg\,min}_{\theta\in\Theta}L(\theta)$). This bound is akin to a local version of strong convexity ($\beta=1$) or a bounded parameter space ($\beta=0$) if $d$ is the Euclidean distance. When $\beta\in[0,1]$, this has also been referred to as the Łojasiewicz assumption introduced in (Łojasiewicz, 1963, 1993). Notably, it has been used in online learning (Gaillard & Wintenberger, 2018) to obtain fast rates with restart strategies. This assumption holds for instance for Example 3.1 with $\beta=1$ (see App C.1). We also discuss this assumption for distributions in the exponential family in Appendix C.2 notably for distributions that have been used practice (Swaminathan & Joachims, 2015b; Kallus & Zhou, 2018; Zenati et al., 2020). Next we state our main result that is the acceleration of the excess risk convergence rate and the regret upper bound of SCRM.

Proposition 5.1**.**

Let $n_{0},n\geq 2$ and $\theta^{*}\in\operatorname*{arg\,min}_{\theta}L(\theta)$. Let $n_{m}=n_{0}2^{m}$ for m=0,\dots,M=\big{\lfloor}\log_{2}(1+\frac{n}{n_{0}})\big{\rfloor}. Then, under Assumptions 4.1, 4.2 and 5.1 with $\beta>0$, the SCRM procedure (Alg. 1) satisfies the excess risk upper-bound

[TABLE]

Moreover, the expected regret is bounded as follows:

[TABLE]

The proof of our result is detailed in Appendix B.3.

Discussion

This result illustrates that an excess risk of order O\big{(}\frac{\log(n)}{n}\big{)} may be obtained when $\beta=1$ (which is implied by a local version of strong convexity assumption in acceleration methods). When $\beta=0$, which merely accounts that the variance of importance weights are bounded, we simply recover the original rate of CRM of order $O(\log(n)/\sqrt{n})$. The SCRM procedures thus improves the excess risk rate whenever $\beta>0$. It is worth to emphasize that the knowledge of $\beta$ is not needed by Alg. 1. We also note that our assumption seems related to the Bernstein condition (Bartlett & Mendelson, 2006, see Def 2.6), and (van Erven et al., 2015, see Def 5.1) that bounds a variance term by an excess risk term to the power. In empirical risk minimization, this implies the same excess risk rate and regret rate (van Erven & Koolen, 2016), which are exactly the same rates as ours (up to logs).

6 Empirical Evaluation

In this section we perform numerical experiments to validate our method in practical settings. We present the experimental setup as well as experiments comparing SCRM to related approaches and internal details of the method.

6.1 Experimental setup

As our method is able to handle both discrete and continuous actions we experiment in both settings. We now provide a brief description of the setups, with extensive details available in Appendix D.2. 111All the code to reproduce the empirical results is available at: https://github.com/criteo-research/sequential-conterfactual-risk-minimization

Continuous actions

We perform evaluation on synthetic problems pertaining to personalized pricing problems from (Demirer et al., 2019) (Pricing) and advertising from (Zenati et al., 2020) (Advertising). We consider Gaussian policies $\pi_{\theta}(\cdot|x)=\mathcal{N}(\mu_{\theta}(x),\sigma^{2})$ with linear contextual parametrization $\mu_{\theta}(x)=\theta^{\top}x$ and fixed variance $\sigma^{2}$ that corresponds to the exploration budget allowed in the original randomized experiment. The features are up to 10 dimensions and the actions are one-dimensional. We keep the original logging baselines from the settings and compare results to a skyline supervised model trained on the whole training data with full information.

Discrete actions

We adapt the setup of (Swaminathan & Joachims, 2015a) that transforms a multilabel classification task into a contextual bandit problem with discrete, combinatorial action space. We keep the original modeling (akin to CRF) with categorical policies $\pi_{\theta}(a|x)\propto\exp(\theta^{\top}(x\bigotimes a))$. The baseline (resp. skyline) is a supervised, full information model with identical parameter space than CRM methods trained on 5% (resp. 100%) of the training data. We consider the class of probabilistic policies that satisfy Assumption 5.1 by predicting actions in an Epsilon Greedy fashion (Sutton & Barto, 1998)): $\pi^{\varepsilon}_{\theta}(a,x)=(1-\varepsilon)\pi_{\theta}(a,x)+\varepsilon/|\mathcal{A}|$ where $\varepsilon=.1$. Real-world datasets include Scene, Yeast and TMC2007 with feature space up to 30,438 dimensions and action space up to $2^{22}$. To account for this combinatorial action space we allow a model $\theta_{m}$ to be learned using data from all past rollouts $\{s_{l}\}_{l<m}$ for better sample efficiency and therefore adjust variance estimation in Appendix A.2 to take into account sequential dependencies.

6.2 SCRM compared to CRM and related methods

We first compare SCRM to CRM and existing methods in the literature.

Comparison between SCRM and CRM

First, we provide insights on the performance that SCRM can achieve compared to classical CRM with increasing sample sizes. The key difference between CRM/SCRM is that for each sample size $n_{m}$ CRM learns from samples generated by the logging model $s^{CRM}_{m}\leftarrow\theta_{0}$ (see Alg. 2) whilst SCRM learns from samples generated by a series of optimized models $s^{SCRM}_{m}\leftarrow\theta_{m}$ (see Alg. 1). For each sample size we select a posteriori the best $\lambda$ for both methods based on test set loss value. We report in Figure 2 over $M=10$ rollouts the mean test loss depending on sample size up to $2^{10}$, with standard deviation estimated over 10 random runs. We observe that SCRM converges very fast, often within the first rollouts. Conversely, CRM needs more samples and the variance is higher. We conclude that there is a striking benefit to use a sequential design in order to achieve near optimal loss with much fewer samples and better confidence compared to CRM. Complementary results on other datasets are available in Appendix E.1.

Moreover, to further illustrate this benefit of efficient learning we also report in Table 1 the sample size needed to attain near optimal performance when $\theta^{*}$ is known as in Example 3.1, where we also observe that SCRM reaches optimal performances faster than CRM. This corroborates the benefits of improved excess risk rates for SCRM.

Hyper-parameter selection for SCRM

In our experiments, hyperparameter selection consists in choosing a value for $\lambda$. We describe a simple heuristic and evaluate its performance on different datasets. We propose to select $\hat{\lambda}_{m}$ by estimating the non-penalized CRM loss (eq. 3) using offline cross-validation on past data $s_{t<m}$. We report in Table 2 the test loss obtained when choosing a fixed $\lambda$ a posteriori ($\lambda^{\prime}$) or with this heuristic ($\hat{\lambda}$). We observe that loss confidence intervals for both methods intersect for all discrete datasets, except on TMC2007 where the degradation shows only at the 3rd digit. On continuous datasets, the heuristic actually improves upon the fixed a posteriori selection. We conclude that this heuristic is usable in practice.

Comparison with other methods

In this paragraph we compare our SCRM to related methods to explore practical implications of existing methods in our setting. We first consider batch bandits methods and implement the stochastic sequential batch pure exploitation (SBPE) algorithm in (Han et al., ) and a batch version of kernel UCB (Valko et al., 2013) algorithm (BKUCB) with an optimized library (see implementations details in Appendix D.3). We also experiment with off-policy RL methods PPO (Schulman et al., 2017) and TRPO (Schulman et al., 2015) from the StableBaselines library (Raffin et al., 2021) (see Appendix D.3). Indeed, such methods model more general state transitions based on past actions, but they could be used in our setting. To fairly compare all methods (in particular those for which no heuristic existing for hyper-parameter selection) we report the mean and standard deviation over 10 random runs of the best test loss a posteriori over hyperparameter grids of the same size. First, we observe that SCRM beats CRM on all datasets, illustrating the benefit of the sequential design. Second, on discrete tasks (where we the combinatorial action space is large) we observe that SCRM achieves nearly the best test loss in all tasks, while RL methods have difficulties maintaining good performances. Third, batch bandits algorithms can achieve good performances in practice because of their deterministic decision rules. However, they involve an $O(n^{3})$ matrix inversion and therefore did not finish (DNF) in 24h (per single run) on a 46 CPU / 500G RAM machine in most of our settings with large sample size $n$, which make them unpractical for large scale experiments. We conclude that SCRM is an effective learning paradigm and that it scales successfully on a variety of settings.

6.3 Details on SCRM

Next, we provide additional empirical evaluations of details of our method.

Evaluation of IPS-IX

To understand the bias-variance trade-off that IPS-IX can achieve in practice compared to other counterfactual estimators we consider a policy evaluation experiment. The task we consider uses sinusoidal losses $y(a)=\cos(a)$ and evaluated policies are shifted Gaussians $\{\pi_{i}=\mathcal{N}(i*\pi/4,1)\}_{i=0,4}$, with $\pi_{0}$ being the logging policy. Evaluated policies with large shifts with $\pi_{0}$ therefore simulate the setting where the logging policy under-explores the action space. The estimators we consider include IPS, SNIPS (Swaminathan & Joachims, 2015b), clipped IPS (eq. 4) with heuristic from (Bottou et al., 2013) and IPS-IX (eq. 9) with $\alpha=1/n$. All methods therefore use their respective heuristics to set hyperparameters. We report in Figure 3 the bias and variance of estimators for each shift $\mu_{0}-\mu=i*\pi/4$ for $i=0,\dots,4$. We observe that IPS-IX shows an empirical bias comparable to IPS, lower than SNIPS and clipped IPS while maintaining a lower variance. Moreover its variance is only slightly higher than clipped IPS which introduced a large bias. We conclude that besides being a key component of our analysis IPS-IX also controls the variance with a better trade-off in practice. More details are available in Appendix E.2.

When is SCRM useful

is a natural question of interest when choosing the method to be used on a given logged bandit feedback problem. Intuitively one can imagine that SCRM will be most useful when the logging policy underexplores the action space, for example when the distance (in parameter space) between the logging and optimal parameters is large. To study this question we proceed to the following experiment on the setup of Example 3.1 with Gaussian distributions $\mathcal{N}(\theta,\sigma)$ and fixed loss variance $\sigma^{*}=\mathrm{Var}_{y}(y)$. We vary the distance $\delta_{0}=\|\theta^{*}-\theta_{0}\|$ between the optimal model $\theta^{*}$ and the logging model $\theta_{0}$. Since the ideal exploration level may be task dependent we choose a posteriori the best $\sigma$ on a grid, for both CRM and SCRM. We report in Figure 4 the best final loss for both CRM and SCRM for a range of values of $\delta_{0}$. We observe in particular that SCRM achieves better final losses for larger distances $\delta_{0}$ than CRM. With the same number of rollouts $M$, SCRM can extend the exploration to further areas while CRM fails for any exploration level in those cases, which advocates for using sequential deployments.

7 Discussions

In this work, we have proposed a method to extend the CRM perspective for designing sequential data collection experiments. We have introduced a novel counterfactual estimator to improve variance control in excess risk bounds. Under a weak error bound assumption, we have sequentially applied these excess risk guarantees to achieve faster rates similarly to acceleration methods. Our method also improves upon CRM in practice and is particularly well-suited for this setting compared to existing methods in the literature. It is worth noting that, in order to avoid introducing dependencies in the excess risk bounds we analyzed, the theoretical algorithm we have studied uses geometric sample sizes to discard previous samples. However, using all past samples has been found to be also effective in practice and developing guarantees for this case would be an interesting area for future research. Additionally, similar to online settings that involve an exploration-exploitation tradeoff, investigating the use of optimism in the face of uncertainty (OFUL) principle in SCRM would also be a promising avenue for future work.

Acknowledgements

The authors thank Alberto Bietti for the insightful early discussions on this project. The authors also thank the reviewers for their feedback on this paper. This work was supported by ANR 3IA MIAI@Grenoble-Alpes (ANR-19-P3IA0003).

Appendix A Additional details on counterfactual estimators

A.1 Unconfoundedness in sequential designs

In these explanations, we recall that the distributions of contexts as well as the distribution of losses are fixed. In other words, the latter do not vary from one batch to another. In the counterfactual risk minimization framework (CRM) (Swaminathan & Joachims, 2015a), the causal graph (using the conventions in (Peters et al., 2017)) can be represented as shown in Figure 5.

In the sequential counterfactual risk minimization (SCRM) framework, if we unfold the causal graph, the following representation can be given in Figure 6.

Therefore, it is clear that in general, $\theta_{t}\not\!\perp\!\!\!\perp\theta_{t+1}$. However, from d-separation and faithfullness (Peters et al., 2017), we have for $t^{\prime}<t$:

[TABLE]

Therefore, given that all the dependencies are observed and that we can condition on the direct parents of a given model $\theta_{t}$, sequential randomized data collection are possible. We eventually provide in Figure 7 an illustration of SCRM and CRM.

A.2 Multiple Importance Sampling Estimators

Note that in order to avoid introducing dependencies in the excess risk bounds we analyzed, the theoretical algorithm we have studied uses geometric sample sizes to discard previous samples. However, using all past samples is effective in practice and developing guarantees for this case would be an interesting area for future research. We present in this section a estimators using aggregation of all previous information. In particular, we can use Multiple Importance Sampling (MIS) (Owen, 2013) over all previous samples. Consider in particular a partition of unity with $m>1$ weight functions $\omega_{t}(a)>0$ which satisfies $\sum_{t=0}^{m}\omega_{t,m}(a)=1$ for all $a$ and $m\in\{0,\dots M\}$. The MIS estimator writes:

[TABLE]

In multiple importance sampling we usually assume that the behavior distributions are independent. In our case, when we optimize $\theta_{t}$ based on the models $\theta_{t-1},\dots,\theta_{0}$, we break this assumption. However, as we will see, we can still have the unbiasedness property and derive an estimator for the variance of the estimator.

Proposition A.1** (Unbiasedness).**

The MIS estimator (12) is unbiased when the loss $y$ is fixed (its distribution $\mathcal{P}_{\mathcal{Y}}(\cdot|x,a)$ does not depend on time rollout $m$).

Proof.

Let $m\in\{1,\dots M\}$. We recall that at all rounds $t<m$, models $\theta_{t}\in\Theta$ were deployed and sets $s_{t}$ of $n_{t}$ observations $s_{t}~{}=~{}(x_{t,i},a_{t,i},l_{t,i},\pi_{t,i})_{i=1,\ldots,n_{t}}$ were collected thereof, with propensities $\pi_{t,i}~{}=~{}\pi_{\theta_{t}}(a_{t,i}|x_{t,i})$ to learn the next model $\theta_{t+1}$. To prove the unbiasedness we use the tower rule on the expectation and condition on previous observations $s_{1},\dots s_{t-1}$:

[TABLE]

where the second last line is true only when the distribution of $y$ does not change over time roll-outs $m$. ∎

Among the proposals for functions $\omega_{t}(a)$, the most ’naive’ and natural heuristic is to choose

[TABLE]

which gives the naive concatenation of all IPS estimators

[TABLE]

where $n=\sum_{t=0}^{m}n_{t}$.

With the previous definition of the empirical mean estimator, we can now derive an empirical variance estimator, starting with the naive multi importance sampling estimator. We write the random variable $r^{m}=({\pi_{\theta}}/{\pi_{\theta_{m}}})y$. We note that for inside a batch $m$ each realization of $r^{m}_{i}=({\pi_{\theta}(a_{m,i}|x_{m,i})}/{\pi_{m,i}})y_{m,i}$ and $r^{m}_{j}$ are independent. But the realizations of the random variables $r^{m}$ and $r^{m^{\prime}}$ are dependent. Writing $n=\sum_{t=0}^{m}n_{t}$

[TABLE]

where the second last equality is obtained with the bilinearity of the covariance. Given the latter expression of the variance, we propose the following estimator and with a linear sampling where all $n_{p}=n_{q}$ for $p,q\in\{1,\dots,M\}$:

[TABLE]

where \hat{V}(r^{m})=\frac{1}{n_{m}(n_{m}-1)}\sum_{i=1}^{n_{m}}\big{(}r_{i}^{m}-\bar{r}^{m}\big{)}^{2} and $\bar{r}^{m}=\frac{1}{n_{m}}\sum_{j=1}^{n_{m}}r_{j}^{m}$.

Note also that for other functions $\omega_{t}(a)$, the most studied one is the balance heuristic with $\omega_{t}\propto n_{t}\pi_{\theta_{t}}(a)$, that is:

[TABLE]

The latter heuristic has been studied for its low variance (Owen, 2013) but these properties have been studied under an i.i.d assumption that is broken in our adaptive data collection strategy. Eventually, note that controlling the variance of this estimator with an implicit exploration estimator as we do in the i.i.d case would make a an interesting research direction.

Appendix B Analysis details

In this section, we provide the details of our analysis by starting with essential definitions, then our proofs of variance dependent excess risk bounds and finally our regret analysis.

B.1 Definitions

$C_{m}(\Theta)$ is a complexity measure that will be upper-bounded by the metric entropy in sup-norm at level $\varepsilon=1/n_{m}$ of the following function set,

[TABLE]

The latter corresponds to clipped prediction errors of policies $\pi_{\theta}$ normalized into $[0,1]$. More precisely, to define rigorously $C_{m}(\Theta)$, we denote for any $n_{m}\geq 1$ and $\varepsilon>0$, the complexity of a class $\mathcal{F}$ by

[TABLE]

where \mathcal{F}\big{(}\{x_{i},a_{i},y_{i}\}\big{)}=\big{\{}\big{(}f(x_{1},a_{1},y_{1}),\dots,f(x_{n},a_{n},y_{n})\big{)},f\in\mathcal{F}\big{\}}\subseteq\mathbb{R}^{n} and the number $\mathcal{H}(\varepsilon,A,\|\cdot\|_{\infty})$ is the smallest cardinality $|A_{0}|$ of a set $A_{0}\subseteq A$ such that $A$ is contained in the finite union of $\varepsilon$-balls centered at points in $A_{0}$ in the metric induced by $\|\cdot\|_{\infty})$. Then, $C_{m}(\Theta)$ is defined by

[TABLE]

B.2 Variance-dependent excess risk bounds

We will denote by $\operatorname{\mathbb{E}}_{m}[\cdot]=\operatorname{\mathbb{E}}[\cdot|s_{0},\dots s_{m}]$ the conditional expectation given the set of observation samples $s_{m}=(x_{m,i},a_{m,i},y_{m,i},\pi_{m,i})_{i=1,\dots,n_{m}}$ up to the rollout $m$. Here, we recall that $x_{m,i}\sim\mathcal{P}_{\mathcal{X}}$, $a_{m,i}\sim\pi_{\theta_{m}}(\cdot|x_{m,i})$, $y_{m,i}\sim\mathcal{P}_{\mathcal{Y}}(\cdot|x_{m,i},a_{m,i})$, and $\pi_{m,i}=\pi_{\theta_{m}}(a_{m,i}|x_{m,i})$. Furthermore, throughout the document, \operatorname{\mathbb{E}}_{x,\theta_{m},y}\big{[}\cdot\big{]} (resp. \mathrm{Var}_{x,\theta_{m},y}\big{[}\cdot]) denotes the expectation (resp. variance) in $(x,a,y)$ where $x\sim\mathcal{P}_{\mathcal{X}}$, $a\sim\pi_{\theta_{m}}(\cdot|x)$, and $y\sim\mathcal{P}_{\mathcal{Y}}(\cdot|x,a)$.

Proposition 4.1 (Generalization Error Bound).

Let $\hat{L}^{\text{IPS-IX}}_{m}$ and $\hat{V}^{\text{IPS-IX}}_{m}$ be the empirical estimators defined respectively in Eq. (9) and Eq. (10). Let $\delta\in(0,1)$, $\theta\in\Theta$, and $n_{m}\geq 2$ the number of samples associated to the logged dataset at round $m$. Then, with probability at least $1-\delta$,

[TABLE]

where $\lambda=\sqrt{18(C_{m}(\Theta)+\log(2/\delta))}$.

Proof.

Let $\delta\in(0,1)$ and $\theta\in\Theta$. Since all functions in $\mathcal{F}_{m,\Theta}$ defined in Eq. (17) take values in $[0,1]$, we can apply the concentration bound of Maurer & Pontil (2009, Theorem 6) to the set $\mathcal{F}_{m,\Theta}$. This yields, with probability at least $1-\delta/2$,

[TABLE]

where

[TABLE]

is an estimation of the sample variance. Let $\alpha>0$ and define the following biased estimator of the excess risk:

[TABLE]

We recall that \operatorname{\mathbb{E}}_{x,\theta_{m},y}\big{[}\cdot\big{]} denotes the expectation in $(x,a,y)$ where $x\sim\mathcal{P}_{\mathcal{X}}$, $a\sim\pi_{\theta_{m}}(\cdot|x)$, and $y\sim\mathcal{P}_{\mathcal{Y}}(\cdot|x,a)$. By construction of $f_{\theta}$ (see Eq. (17)),

[TABLE]

where $\hat{L}^{\text{IPS-IX}}_{m}$ and $\hat{V}^{\text{IPS-IX}}_{m}$ are defined respectively in Eq. (9) and Eq. (10). Thus, multiplying (21) by $W$, substituting the above terms, and using $\lambda=\sqrt{18(C_{m}(\Theta)+\log(2/\delta))}$, yields

[TABLE]

with probability $1-\delta/2$. Now, let us decompose

[TABLE]

But, since the losses $y$ are bounded in $[-1,0]$ almost surely,

[TABLE]

which, substituted into the previous equation, entails,

[TABLE]

Lower-bounding the left-hand side of (26), we thus get w.p $1-\delta/2$,

[TABLE]

Using $\operatorname{\mathbb{E}}_{m-1}[y_{m,i}]=L(\theta_{m})$ and applying Hoeffding’s inequality, this further yields w.p. $1-\delta$

[TABLE]

Eventually, note that $(n_{m}-1)^{-1}\leq({2}/{n_{m}})$ since $n_{m}\geq 2$. Thus,

[TABLE]

which concludes the proof.

∎

Proposition 4.2 (Conservative Excess Risk).

Let $m\geq 0$ and $\theta_{m}\in\Theta$. Let $s_{m}=(x_{m,i},a_{m,i},y_{m,i},\pi_{m,i})_{1\leq i\leq n_{m}}$ be a set of samples collected with $a_{m,i}\sim\pi_{\theta_{m}}(\cdot|x_{m,i})$. Then, under Assumptions 4.1 and 4.2, the solution $\theta_{m+1}$ of Problem (8) with the IPS-IX estimator in Eq. (11) on the samples $s_{m}$ satisfies the excess risk upper-bound

[TABLE]

where $\nu_{m}^{2}=\mathrm{Var}_{x,\theta_{m}}\left(\dfrac{\pi_{\theta^{*}}(a|x)}{\pi_{\theta_{m}}(a|x)}\right)$.

Proof.

We consider the notations of the proof of Proposition 4.1. Fix $\theta^{*}\in\Theta$. Applying, Theorem 15 of (Maurer & Pontil, 2009)222Note that in their notation, $\log\mathcal{M}_{n}(\pi)$ equals $C_{m}(\Theta)+\log(10)$, $\bf X$ is the dataset $\smash{\{(x_{i},a_{i},y_{i})\}_{1\leq i\leq n}}$ where $\smash{(x_{i},a_{i},y_{i})\stackrel{{\scriptstyle i.i.d.}}{{\sim}}\mathcal{P}_{\mathcal{X}}\times\pi_{\theta_{m}}(\cdot|x)\times\mathcal{P}_{\mathcal{Y}}(\cdot|a,x)}$, and $P(\cdot,\mu)$ is the expectation with respect to one test sample $\operatorname{\mathbb{E}}_{x,\theta_{m},y}[\,\cdot\,]$. to the function set $\mathcal{F}_{m,\Theta}$ defined in (17), we get with probability $1-\delta$

[TABLE]

This can be written as:

[TABLE]

with the following definitions:

[TABLE]

Step: Lower bounding $\Delta^{*}_{m}$

Using the definition of $f_{\theta}(x,a,y)$ in (17) and that of $L^{\alpha}_{m}$ in Eq. (22), we have

[TABLE]

Thus, $\Delta_{m}^{*}$ can be re-written as

[TABLE]

which we now lower-bound. To do so, we begin by upper-bounding $L^{\alpha}_{m}(\theta^{*})$. It can be expressed as

[TABLE]

To shorten notation, from now on and throughout this proof, we write $\pi_{\theta}$ instead of $\pi_{\theta}(a|x)$, omitting the dependence on $a$ and $x$. Using the inequality $(1+x)^{-1}\geq 1-x$ for $x\geq 0$, we have

[TABLE]

where the last inequality is by Assumption 4.1 and because $y\in[-1,0]$. Together with (30), we get

[TABLE]

We recall that $L(\theta_{m+1})-L(\theta_{m})\leq L^{\alpha}_{m}(\theta_{m+1})$ by Eq.(24). Therefore,

[TABLE]

which finally gives

[TABLE]

Step: Upper bound $U^{*}_{m}$

By definition of $f_{\theta}(x,a,y)$ in (17), we have

[TABLE]

Then, using the inequality $(x+y)^{2}\leq 2x^{2}+2y^{2}$, for $x,y\in\mathbb{R}$, this may be upper-bounded as

[TABLE]

On the one hand, the first term of the right-hand side may be upper-bounded as

[TABLE]

where $\nu_{m}^{2}=\mathrm{Var}_{x,\theta_{m}}\left[\frac{\pi_{\theta^{*}}}{\pi_{\theta_{m}}}\right]$. On the other hand, for the second term, we use the same factorization as in Eq. (31) to get

[TABLE]

which yields the upper-bound

[TABLE]

Therefore, substituting the last two upper-bounds into (34) entails

[TABLE]

Then, replacing this upper-bound into the definition of $U_{m}^{*}$ in (29) and using Assumption 4.2 to upper bound the terms in $C_{m}(\Theta)\leq d\log(n_{m})$, we obtain the following upper-bound

[TABLE]

where the last inequality is because $n_{m}\geq 2$.

Step: excess risk upper bound

Setting $\alpha=\frac{1}{n_{m}}$ and using the two previous bounds (33) and (35) respectively on $\Delta^{*}_{m}$ and on $U^{*}_{m}$ into (28), we get

[TABLE]

Using that $\sqrt{a+b}\leq\sqrt{a}+\sqrt{b}$, we have that

[TABLE]

Then, since $n_{m}\geq 2$ and $\delta<1$, we have $d\log(n_{m})+\log(30/\delta)\geq\log(2)+\log(30)\geq 4$, which yields

[TABLE]

Substituting the last two inequalities into (36) finally entails

[TABLE]

which concludes the proof. ∎

B.3 Regret analysis

Proposition 5.1 (Regret upper-bound).

Let $n_{0},n\geq 2$ and $\theta^{*}\in\operatorname*{arg\,min}_{\theta}L(\theta)$. Let $n_{m}=n_{0}2^{m}$ for m=0,\dots,M=\big{\lfloor}\log_{2}(1+\frac{n}{n_{0}})\big{\rfloor}. Then, under Assumptions 4.1, 4.2 and 5.1, the SCRM procedure (Alg. 1) satisfies the excess risk upper-bound

[TABLE]

Moreover, the expected regret is upper-bounded as follows:

[TABLE]

Proof.

First, note that for $n_{m}=n_{0}2^{m}$ and M=\big{\lfloor}\log_{2}(1+\frac{n}{n_{0}})\big{\rfloor}, we have $\sum_{m=0}^{M-1}n_{m}=n_{0}(2^{M}-1)\leq n.$ Hence, Alg. 1 has collected at most $n$ samples to design the estimator $\theta_{M}$. For $m\geq 0$, we recall $\Delta_{m}=L(\theta_{m})-L(\theta^{*})$ and use Eq. (37) to write

[TABLE]

where $C=8\sqrt{d\log(n)+\log\frac{30}{\delta}}$ and $B=W^{2}+47W(d\log(n)+\log\frac{30}{\delta})$ are independent of $m$.

Step: Obtaining a recurrence relation for $\Delta_{m+1}$

By Assumption 5.1, there exist $\gamma>0$ and $\beta\in[0,1]$ such that

[TABLE]

Replacing $\nu_{m}^{2}$ in Eq. (38) thus entails

[TABLE]

Step: Solving the recurrence relation for $\Delta_{m}$

We then insure by induction that $\Delta_{m}$ satisfies

[TABLE]

for some $c_{0}>0$ that will be specified by the analysis.

Base step Since losses take values in $[-1,0]$, $\Delta_{0}=L(\theta_{0})-L(\theta^{*})\leq 1$. Equation (40) is thus satisfied for $m=0$ as soon as $c_{0}\geq 1$.

Induction step Let $m\geq 0$. We assume that $\Delta_{m}\leq c_{0}2^{\frac{-m}{2-\beta}}$ and prove Equation (40) for $\Delta_{m+1}$. Using Eq. (39), we have

[TABLE]

Now, we show that both terms inside the maximum can be upper-bounded by $c_{0}2^{-(m+1)/(2-\beta)}$ as soon as $c_{0}$ is large enough. On the one hand, if $c_{0}\geq 4Bn_{0}$, we have

[TABLE]

On the other hand, if $c_{0}\geq(4C^{2}n_{0}/\gamma)^{1/(2-\beta)}$, we also have

[TABLE]

Combining the above two upper-bounds with (41) concludes the induction step under the condition

[TABLE]

Step: conclusion

Finally, setting the above value for $c_{0}$ we proved that for all $m\geq 0$, we have

[TABLE]

where the last equality is by substituting the values of $B$ and $C$ from (38). For the final step $M~{}=~{}\lfloor\log_{2}(\frac{n}{n_{0}}+1)\rfloor$, this yields

[TABLE]

This concludes the first part of the proof.

Regret upper-bound

To upper bound the cumulative regret, using $n_{m+1}=n_{0}2^{m+1}$, we write

[TABLE]

where

[TABLE]

Then, computing the sum for $M~{}=~{}\lfloor\log_{2}(\frac{n}{n_{0}}+1)\rfloor$, we have

[TABLE]

Using that $D=O(\log n)$, we finally obtain

[TABLE]

∎

Appendix C Additional discussions on the Hölderian Bound Assumption 5.1

In this appendix, we discuss Assumption 5.1 on different particular examples.

C.1 Verification of the assumption on a toy example with Gaussian families

We consider the setting of Example 3.1. In the latter, the policies are Gaussian of the form $\pi_{\theta}=\mathcal{N}(\theta,\sigma^{2})$ and the loss is defined by $l_{t}(a)=(a-y_{t})^{2}-1$ where $y_{t}\sim\mathcal{N}(\theta^{*},\sigma^{2})$. There is no loss in generality in assuming $\sigma^{2}=1$. Then, we can compute

[TABLE]

We recall that we are interested in verifying the existence of $\gamma>0$ and $\beta>0$ for which Assumption 5.1 holds, that is in this case for any $\theta\in\Theta$:

[TABLE]

which may be re-written here as

[TABLE]

The latter is satisfied for any $\beta\leq 1$ as soon as $\Theta$ is a bounded interval. Note that the constant $\gamma$ may decrease exponentially fast as the diameter of $\Theta$ increases. To illustrate, the existence of such couples $(\beta,\gamma)$, we plot in Fig. 8 different values of the following ratio

[TABLE]

The value of $\gamma$ can be found for different values of $\beta$ in Fig. 8 by taking $\frac{1}{\gamma}=\max_{\theta}R(\theta,\beta)$.

Higher values of $\beta$ induce faster rates and lower values of $\gamma$ induce worst constant terms in the excess risk and regret bounds. Eventually, note that SCRM does not need those parameters to run and those two parameters $\gamma,\beta$ are automatically calibrated by SCRM to find the best trade-off.

C.2 Discussion of Assumption 5.1 for Exponential Families

In this section, we consider a more realistic example in which policies belong to an exponential family. That is, we assume that the policies are parameterized by a parameter $\eta\in\mathbb{R}^{q}$ and can be written in the form:

[TABLE]

for some known function $h:\mathcal{A}\to\mathbb{R}_{+}$ and sufficient statistic $t:\mathcal{A}\to\mathbb{R}^{q}$. Here, $A(\eta)$ is a normalization constant, so that $e^{A(\eta)}=\int_{a}e^{\eta\cdot t(a)}h(a)\mathop{}\!\mathrm{d}a$. We provide in Example C.1 a concrete example considered by (Swaminathan & Joachims, 2015a; Faury et al., 2020). To ease the notation, we removed here the dependency on contexts, but the generalization to contextual policies can be made similarly. The importance weight ratio may be written as,

[TABLE]

To verify Assumption 5.1, we need to upper bound their variance, which we shall write as,

[TABLE]

Now, computing the moment generating function (MGF) of the statistic $t(a)\in\mathbb{R}^{q}$

[TABLE]

the variance term may be written as

[TABLE]

This eventually leads us to

[TABLE]

We now discuss two cases that are used for discrete actions (Swaminathan & Joachims, 2015a) and continuous actions (Kallus & Zhou, 2018; Zenati et al., 2020).

Bounded sufficient statistic

Supposing that there exists an upper bound $A$ such that $\|t(a)\|\leq A$, Cauchy-Schwartz inequality states that $|(\eta-\eta_{m})\cdot t(a)|\leq\|\eta-\eta_{m}\|A$, which entails

[TABLE]

Assuming that the parameter space is compact, i.e, $\max_{\eta,\eta^{\prime}}\|\eta-\eta^{\prime}\|\leq D$, there exists a constant $C$ that depends on $A$ and $D$ such that, this may be further upper-bounded as

[TABLE]

Therefore, Assumption 5.1 is implied by

[TABLE]

The latter is implied by a local version of strong convexity for $\beta=1/2$ (d’Aspremont et al., 2021), and holds with $\gamma=C^{-1}D^{-2}$ for $\beta=0$.

Example C.1**.**

For discrete actions $\mathcal{A}=\{a_{1},\dots,a_{K}\}$, we consider, as in (Swaminathan & Joachims, 2015a) and (Faury et al., 2020), policies where given a context x, probabilities $p_{i}(x)$ of sampling an action $a_{i}$ are given by

[TABLE]

The function $\phi$ is typically a feature map associated to a kernel in a RKHS. In this case, the natural parameter $\eta$ and the sufficient statistic $t(a)$ may be written as

[TABLE]

Lognormal and Normal distributions

For normal $\mathcal{N}(\mu,\sigma^{2})$ and lognormal $\text{Lognormal}(\mu,\sigma^{2})$ distributions with fixed variance $\sigma^{2}$ as considered by (Kallus & Zhou, 2018; Zenati et al., 2020), the normalizing constant writes $A(\eta)=\frac{\eta^{2}}{2}$, and we then obtain that:

[TABLE]

which gives:

[TABLE]

In that case, it is again possible for a bounded parameter space to linearize $e^{\|\eta-\eta_{m}\|^{2}}-1\lesssim\|\eta-\eta_{m}\|^{2}$, consider losses that verify: for all $\eta$, there exists an optimal $\eta^{*}$ such that

[TABLE]

Again, this holds generally for $\beta=0$ and for locally strongly convex losses for $\beta=1$.

Appendix D Experiment details

D.1 Code

All the code to reproduce figures and tables is available in the following repository: https://github.com/criteo-research/sequential-conterfactual-risk-minimization.

D.2 Empirical settings details

Pricing

The pricing application in (Demirer et al., 2019) considers a ”personalized pricing” setting where given contexts $x$, prices $p$ (which are the actions) need to be predicted to maximize the revenue:

[TABLE]

where $\varepsilon\sim\mathcal{N}(0,1)$ and $d=a(x)+b(x)p+\varepsilon$ is akin to an unknown context-specifidemand function. The data generating process uses contexts $x\in[1,2]^{k}$ for $k>1$ a positive integer. Only $l<k$ dimensions however affect the demand, that is if we write $\bar{x}=\frac{1}{l}(z_{1},\dots,z_{l})$. The price $p$ is generated from a Gaussian logging policy $p\sim\mathcal{N}(\bar{x},1)$ centered in $\bar{x}$. We consider in our example the quadratic functionnal $a(x)=2x^{2}$ and $b(x)=0.6x$ as in the original paper.

Advertising

The advertising simulation in (Zenati et al., 2020) consists in predicting the potential $p\in]0,+\infty[$ of a user that may be compared to their a priori responsiveness to a treatment. The potential is caused by an unobserved random group variable $g$ in $G$ (groups of ”high” or ”low” potential users in their responsiveness) that influences context $x$ of users. The goal is then to find a policy $\pi(a|x)$ that maximizes reward by adapting to an unobserved potential. The potentials are normally distributed conditionally on the group index, $p|g\sim\mathcal{N}(\mu_{g},\sigma_{g}^{2})$ where $\sigma_{g}=0.5$ and $\mu_{g}=1$ or $3$ for two groups. The observed reward $-y$ is then a function of the action $a$ and the context $x$ through the associated potential $p_{x}$ of the user $x$. The reward function mimics reward over the offline continuous bidding dataset in (Zenati et al., 2020) with the form:

[TABLE]

The logging policy is a lognormal distribution as it is common in advertising applications (Bottou et al., 2013). In particular, as in (Zenati et al., 2020), $\pi_{\theta_{0}}=\text{Lognormal}(\mu,\sigma^{2})$ where the mean $\exp(\mu+{\sigma^{2}}/{2})=2$ and the variance $(\exp(\sigma^{2})-1)\exp(2\mu+\sigma^{2})=1$.

Yeast, Scene, TMC2007

We follow (Swaminathan & Joachims, 2015a). We now recall briefly the setup. The problem is a binary multilabel classification with $|\mathcal{A}|=2^{K}$ potential labels. All models are parametrized by $\pi_{\theta}(a|x)\propto\exp(\theta^{\top}(x\bigotimes a))$. The baseline (resp. skyline) is a supervised, full information model with identical parameter space than CRM methods trained on 5% (resp. 100%) of the training data. Our main modification it to consider the class of probabilistic policies that satisfy Assumption 5.1 by predicting actions in an Epsilon Greedy fashion (Sutton & Barto, 1998)): $\pi^{\varepsilon}_{\theta}(a,x)=(1-\varepsilon)\pi_{\theta}(a,x)+\varepsilon/|\mathcal{A}|$ where $\varepsilon=.1$. The loss is the Hamming loss (number of incorrectly assigned labels - both false positives and false negatives in the action vector):

[TABLE]

where $y_{i}^{j}$ (resp. $a_{i}^{j}$) is the $j$-th component of the label vector (resp. action vector) of line $i$. A uniform policy will thus evaluate at a loss of $.5$.

D.3 Implementation details

Counterfactual methods

In this paragraph we start by detailing the non adaptive counterfactual risk minimization that we compare to in this work.

We also provide the grid of hyperparameters for the $\lambda$ evaluated in CRM and SCRM methods $\lambda\in[1e-5,1e-4,1e-3,1e-2,1e-1]$.

Batch Bandits

Let $k:(\mathcal{X}\times\mathcal{A})\times(\mathcal{X}\times\mathcal{A})\to\mathbb{R}$ be a bounded positive definite Kernel associated to a RKHS $\mathcal{H}$, $\phi:\mathcal{X}\times\mathcal{A}\to\mathcal{H}$ is the feature map such that $k(s,s^{\prime})=\langle\phi(s),\phi(s^{\prime})\rangle$ for any $s,s^{\prime}\in\mathcal{X}\times\mathcal{A}$. Context-actions pairs are written as $s_{m,i}\vcentcolon=(x_{m,i},a_{m,i})\in\mathcal{X}\times\mathcal{A}$ and $\mathcal{S}_{m}\vcentcolon=\{s_{1,0},\dots,s_{n_{m},m}\}$ denoting the history of all context-actions pairs seen up until the end of batch $m$. $K_{m}$ is the kernel matrix of all context-actions seen until the end of the batch $m\geq 1$. Eventually, $K_{\mathcal{S}}(s^{\prime})$ is the kernel column vector $[k(s_{1},s^{\prime}),\dots,k(s_{l},s^{\prime})]^{\top}$ of size $|\mathcal{S}|=l$. $Y_{m}=[-y_{0,1},\cdots-y_{0,n_{0}},\cdots-y_{m,1},\cdots-y_{m,n_{m}}]$ denotes the vector of concatenated rewards observed up until the end of the batch $m$.

At a batch $m$, a context $x_{m,i}$ is sampled for $i\in\{1,n_{m}\}$, and then to sample an action $a$, the following decision rule is applied:

[TABLE]

In batch Kernel UCB, $\hat{q}_{m,i,a}$ is defined as

[TABLE]

where

[TABLE]

and $\beta_{m}$ is a theoretical parameter that is set to $\beta_{m}=\frac{1}{\sqrt{m}}$ in practical heuristics (Lattimore & Szepesvari, 2019). In SBPE (Han et al., ), $\hat{q}_{m,i,a}$ is defined directly as

[TABLE]

SBPE (Han et al., ) uses a linear modelling, therefore we used a linear kernel. For the Kernel UCB (Valko et al., 2013) method, we used Gaussian and Polynomial kernels in our experiments. Note also that no regularization parameter $\lambda$ is used in SBPE so we set $\lambda=0$ in our experiments, and for K-UCB we chose $\lambda$ in the grid $[1e0,1e1,1e2]$.

Note in particular that we adapted the batch bandit baselines to the CRM setting by benefiting the initialization with the logged dataset to set the gram matrix $K_{\lambda}$ as well as the reward vector $Y_{0}$ with information from the logging data. This modification changes the original methods which take random actions at initializations.

Eventually, the baselines were carefully optimized using the Jax library (https://github.com/google/jax) to allow for just in time compilations of algebraic blocks in both methods and to maximize their scaling capacity.

RL baselines

In order to compare our method to the two known off-policy online RL algorithm PPO (Schulman et al., 2017) and TRPO (Schulman et al., 2015), we do the following:

we use the stable_baselines3(Raffin et al., 2021) library for the implementation. When necessary we call multiple times the model PPO or TRPO, to have buffer size of geometrical increase. 2. 2.

we initialize the ActorCriticPolicy with a simpler MLP model having only one layer with output dimension of 1, (with argument net_arch= [1], that is mathematically the same modelling as in CRM and SCRM baselines). 3. 3.

At the initial step only and to enable a fair comparison with counterfactual methods using a logging dataset, we pretrain the RL policies to imitate the actions sampled from the logging policy: we process by multiple step of the Adam optimizer, minimizing a loss being the sum of 2 terms:

• •

a MSE term between the sampled action of the ActorCriticPolicy for the contexts in the $n_{0}$ instances, and the actions sampled by the logging policy.

• •

the ENTROPY term guaranteeing to keep a minimum of exploration in order to initialize the RL algorithm ($-\sum p_{i}\log(p_{i})$) 4. 4.

we combine the 2 last terms with a linear combinaison with hyperparameters being tuned a posteriori, i.e. $\text{LOSS}=\text{MSE}+\lambda\text{ ENTROPY}$ with the hyperparam $\lambda\in\{.5,1,2,5,10\}$

Appendix E Additional empirical results

E.1 SCRM compared to CRM

We provide here the additional plot in the Pricing setting.

E.2 Evaluation of IPS-IX

We provide here the plots for the whole setting considered in policy evaluation with IPS-IX.

E.3 Exploration/Exploitation tradeoff

In this part we give the details used for the experiment described in Section 6.3. We consider again Example 3.1 with the Gaussian parametrized policies $\pi_{\theta}=\mathcal{N}(\theta,\sigma^{2})$ and a loss $l_{t}(a)=(a-y_{t})^{2}-1$ where $y_{t}\sim\mathcal{N}(\theta^{*},{\sigma^{*}}^{2})$ with $\sigma^{*}=0.3$. Recall that $\pi_{\theta_{0}}=\mathcal{N}(\theta_{0},\sigma)$. We consider a grid of $\sigma\in[0.1,0.3,1,3]$ and consider $\theta^{*}=1$. Our experiment aims at illustrating the influence of sequential exploration that is an important detail of the SCRM and CRM principles.