Causal Embeddings for Recommendation

TL;DR

This paper introduces a causal embedding approach for recommendation systems that optimizes for desired outcomes by learning from biased logged data, bridging the gap between traditional relevance metrics and actual business goals.

Contribution

The paper proposes a novel domain adaptation algorithm that predicts recommendation outcomes under random exposure using logged data from biased policies.

Findings

Significant improvement over state-of-the-art factorization methods.

Effective learning from biased logged data for outcome prediction.

Enhanced recommendation policy optimization for business objectives.

Abstract

Many current applications use recommendations in order to modify the natural user behavior, such as to increase the number of sales or the time spent on a website. This results in a gap between the final recommendation objective and the classical setup where recommendation candidates are evaluated by their coherence with past user behavior, by predicting either the missing entries in the user-item matrix, or the most likely next event. To bridge this gap, we optimize a recommendation policy for the task of increasing the desired outcome versus the organic user behavior. We show this is equivalent to learning to predict recommendation outcomes under a fully random recommendation policy. To this end, we propose a new domain adaptation algorithm that learns from logged data containing outcomes from a biased recommendation policy and predicts recommendation outcomes according to random exposure. We compare our method against state-of-the-art factorization methods, in addition to new approaches of causal recommendation and show significant improvements.

Figures

Tables

Table 1. Table 1 . Definitions and Notations

Symbol	Definition
$u_i$	A user of the recommendation system
$p_j$	A product which the system can recommend
${\pi }_x$	A recommendation policy (eq. 1)
${\pi }_c$	The control recommendation policy. This represents the recommendation system used to create the training dataset.
${\pi }_t$	The treatment recommendation policy. This represents the updated recommendation system.
${\pi }^{rand}$	The fully random recommendation policy that shows any product with equal probability to all users.
$r_{ij}$	The true reward for recommending a product $p_j$ to user $u_i$
$y_{ij}$	The observed reward for recommending product $p_j$ to user $u_i$ in the data. By comparison with $r_{ij}$, its value can be unknown.
$R^{{\pi }_x}$	The total reward for policy ${\pi }_x$ (eq. 2)
$IT{E}_{ij}^{{\pi }_x}$	The difference between the reward for current and control policy (eq. 3)
$p_i^{\ast }$	The product with the highest reward for user $u_i$ (eq. 6)
${\pi }^{\ast }$	The best incremental recommendation policy (eq. 4)
$S_c$	A large set of training samples collected under the control recommendation policy
$S_t$	A smaller set of samples taken under a full-randomized recommendation policy

Table 2. Table 2 . Results for MovieLens10M and Netflix on the Skewed (SKEW) test datasets. All three versions of the CausE algorithm outperform both the standard and the IPS-weighted causal factorization methods, with CausE-avg and CausE-prod-C also out-performing BanditNet. We can observe that our best approach CausE-prod-C outperforms the best competing approaches WSP2V-blend by a large margin (21% MSE and 20% NLL lifts on the MovieLens10M dataset) and BN-blend (5% AUC lift on MovieLens10M).

Method	MovieLens10M (SKEW)			Netflix (SKEW)
	MSE lift	NLL lift	AUC	MSE lift	NLL lift	AUC
BPR-no	$-$	$-$	$0.693\left(\pm 0.001\right)$	$-$	$-$	$0.665\left(\pm 0.001\right)$
BPR-blend	$-$	$-$	$0.711\left(\pm 0.001\right)$	$-$	$-$	$0.671\left(\pm 0.001\right)$
SP2V-no	$+3.94\%\left(\pm 0.04\right)$	$+4.50\%\left(\pm 0.04\right)$	$0.757\left(\pm 0.001\right)$	$+10.82\%\left(\pm 0.02\right)$	$+10.19\%\left(\pm 0.01\right)$	$0.752\left(\pm 0.002\right)$
SP2V-blend	$+4.37\%\left(\pm 0.04\right)$	$+5.01\%\left(\pm 0.05\right)$	$0.768\left(\pm 0.001\right)$	$+12.82\%\left(\pm 0.02\right)$	$+11.54\%\left(\pm 0.02\right)$	$0.764\left(\pm 0.003\right)$
SP2V-test	$+2.45\%\left(\pm 0.02\right)$	$+3.56\%\left(\pm 0.02\right)$	$0.741\left(\pm 0.001\right)$	$+05.67\%\left(\pm 0.02\right)$	$+06.23\%\left(\pm 0.02\right)$	$0.739\left(\pm 0.004\right)$
WSP2V-no	$+5.66\%\left(\pm 0.03\right)$	$+7.44\%\left(\pm 0.03\right)$	$0.786\left(\pm 0.001\right)$	$+13.52\%\left(\pm 0.01\right)$	$+13.11\%\left(\pm 0.01\right)$	$0.779\left(\pm 0.001\right)$
WSP2V-blend	$+6.14\%\left(\pm 0.03\right)$	$+8.05\%\left(\pm 0.03\right)$	$0.792\left(\pm 0.001\right)$	$+14.72\%\left(\pm 0.02\right)$	$+14.23\%\left(\pm 0.02\right)$	$0.782\left(\pm 0.002\right)$
BN-blend	$-$	$-$	$0.794\left(\pm 0.001\right)$	$-$	$-$	$0.785\left(\pm 0.001\right)$
CausE-avg	$+12.67\%\left(\pm 0.09\right)$	$+15.15\%\left(\pm 0.08\right)$	$0.804\left(\pm 0.001\right)$	$+15.62\%\left(\pm 0.02\right)$	$+15.21\%\left(\pm 0.02\right)$	$0.799\left(\pm 0.002\right)$
CausE-prod-T	$+07.46\%\left(\pm 0.08\right)$	$+10.44\%\left(\pm 0.09\right)$	$0.779\left(\pm 0.001\right)$	$+13.97\%\left(\pm 0.02\right)$	$+13.52\%\left(\pm 0.02\right)$	$0.789\left(\pm 0.003\right)$
CausE-prod-C	$+15.48\%\left(\pm 0.09\right)$	$+19.12\%\left(\pm 0.08\right)$	$0.814\left(\pm 0.001\right)$	$+17.82\%\left(\pm 0.02\right)$	$+17.19\%\left(\pm 0.02\right)$	$0.821\left(\pm 0.003\right)$

Equations

$$p_j\sim {\pi }_x(.|u_i)$$

$$r_{ij}\sim r(.|u_i,p_j)$$

$$y_{ij}=r_{ij}{\pi }_x(p_j|u_i)$$

$$R^{{\pi }_x}=\sum _{ij}{r}_{ij}{\pi }_x(p_j|u_i)p(u_i)=\sum _{ij}{y}_{ij}p(u_i)=\sum _i{R}_{ij}$$

$$IT{E}_{ij}^{{\pi }_x}=R_{ij}^{{\pi }_x}-R_{ij}^{{\pi }_c}$$

$${\pi }^{\ast }=arg\underset{{\pi }_x}{\max }\{IT{E}^{{\pi }_x}\}$$

$${\pi }^{\ast }=arg\underset{{\pi }_x}{\max }\sum _{ij}{r}_{ij}{\pi }_x(p_j|u_i)p(u_i)$$

$$p_i^{\ast }=arg\underset{p_j}{\max }{r}_{ij}$$

$$\begin{array}{r}{R}^{{\pi }_x}=\sum _{i}p(u_i)\sum _j{r}_{ij}{\pi }_x(p_j|u_i)\le \\ \le \sum _{i}p(u_i)\sum _j{r}_i^{\ast }{\pi }_x(p_j|u_i)=\sum _{i}p(u_i)r_i^{\ast }=R^{{\pi }_{det}}\end{array}$$

$${\pi }_{det}=\{\begin{array}{ll}1,& \text{if }{p}_j=p_i^{\ast }\\ 0,& \text{otherwise}\end{array}$$

$${\hat{r}}_{ij}\approx \frac{y_{ij}}{{\pi }_c(p_j|u_i)}$$

$$min\sum _{u_i}p(u_i)L(\widehat{p_i},p_i^{\ast })$$

$$\begin{array}{r}\widehat{p_i}=arg\underset{p_j}{\max }{\hat{r}}_{ij}=arg\underset{p_j}{\max }\frac{{\hat{y}}_{ij}}{{\pi }_c(p_j|u_i)}=\\ =arg\underset{p_j}{\max }\frac{{\hat{y}}_{ij}^{rand}}{{\pi }^{rand}(p_j|u_i)}=arg\underset{p_j}{\max }{\hat{y}}_{ij}^{rand}\end{array}$$

$$\hat{w}=arg\underset{w}{\min }\frac{1}{n}\sum _{i=1}^n({\delta }_i-\lambda )\frac{{\pi }_w(p_j|u_i)}{{\pi }_c(p_j|u_i)}$$

$$\begin{array}{r}{y}_{ij}^c\approx <{\theta }_j^c,u_i>\\ y_{ij}^t\approx <{\theta }_j^t,u_i>\end{array}$$

$${\widehat{ITE}}_{ij}=<{\theta }_j^t,u_i>-<{\theta }_j^c,u_i>=<{\theta }_j^{\Delta },u_i>$$

$$l_{ij}^t=L(<{\theta }_j^t,u_i>,y_{ij}^t)+\Omega ({\theta }_j^t)$$

$$L_t=\sum _{(i,j,y_{ij})\in S_t}{l}_{ij}^t=L(U{\Theta }_t,Y_t)+\Omega ({\Theta }_t)$$

$$l_{ij}^c=L(<{\theta }_j^t-{\theta }_j^{\Delta },u_i>,y_{ij}^c)+\Omega ({\theta }_j^t)+\Omega ({\theta }_j^{\Delta })$$

$$\begin{array}{r}{L}_c=L(U{\Theta }_c,Y_c)+\Omega ({\Theta }_c)+\Omega ({\Theta }_t-{\Theta }_c)\end{array}$$

$$\begin{array}{r}{L}_{CausE}^{prod}=\underset{treatmenttaskloss}{\underbrace{L(U{\Theta }_t,Y_t)+\Omega ({\Theta }_t)}}+\underset{controltaskloss}{\underbrace{L(U{\Theta }_c,Y_c)+\Omega ({\Theta }_c)}}+\\ +\underset{regularizerbetweentasks}{\underbrace{\Omega ({\Theta }_t-{\Theta }_c)}}\end{array}$$

$$\begin{array}{r}{L}_{CausE}=\underset{treatmenttaskloss}{\underbrace{L({\Gamma }_t{\Theta }_t,Y_t)+\Omega ({\Gamma }_t,{\Theta }_t)}}+\underset{controltaskloss}{\underbrace{L({\Gamma }_c{\Theta }_c,Y_c)+\Omega ({\Gamma }_c,{\Theta }_c)}}+\\ +\underset{regularizersbetweentasks}{\underbrace{\Omega ({\Gamma }_t-{\Gamma }_c)+\Omega ({\Theta }_t-{\Theta }_c)}}\end{array}$$

$$lif{t}_x^{metric}=\frac{metri{c}_x-metri{c}_{AvgCR}}{metri{c}_{AvgCR}}$$

$${\hat{y}}_{ij}=\sigma (\alpha <p_j,u_i>+b_i+b_j+b)$$

Full text

Recommender Systems; Causality; Embeddings; Neural Networks; Counterfactual Inference

††copyright: rightsretained††journalyear: 2018††copyright: acmlicensed††conference: Twelfth ACM Conference on Recommender Systems; October 2–7, 2018; Vancouver, BC, Canada††booktitle: Twelfth ACM Conference on Recommender Systems (RecSys ’18), October 2–7, 2018, Vancouver, BC, Canada††price: 15.00††doi: 10.1145/3240323.3240360††isbn: 978-1-4503-5901-6/18/10††ccs: Computing methodologies Learning from implicit feedback††ccs: Computing methodologies Neural networks††ccs: Computing methodologies Transfer learning

1. Introduction

In recent years, online commerce has outpaced the growth of traditional business. The research work on recommender systems has consequently grown significantly during the same time-frame. As shown by key players in the online e-commerce space, such as Amazon, YouTube or Netflix, the product recommendation functionality is now a key driver of demand, accounting for, in the case of Amazon, roughly $35\%$ (Marshall, 2006) of overall sales.

Within the area of recommender systems, a promising new class of deep-learning based solutions has emerged and are showing promising results. We can separate the newly proposed solutions in two main types of approaches: ones that learn item embeddings that optimize for item-item similarity prediction (Grbovic et al., 2015; Vasile et al., 2016; Pennington et al., 2014) and the ones that learn user sequence embeddings to optimize for next item prediction (Hidasi et al., 2015, 2016). The new methods can scale to millions of users and items and show good performance improvements over traditional approaches. In the context of product recommendation at scale, these deep-learning approaches were successfully applied to ad recommendations in Yahoo! Mail (Grbovic et al., 2015), for video recommendation at YouTube (Covington et al., 2016) and restaurant recommendations by OpenTable (Das, 2015).

However, even with the recent model developments, the current state-of-the-art machine learning models still frame the recommendation task as either:

• •

A distance learning problem between pairs of products or between pairs of users and products, where we measure outcomes using link prediction/matrix completion metrics such as Mean Squared Error (MSE) and Area Under the Curve (AUC).

• •

A next item prediction problem that models the user behavior and tries to predict what she will do next, using ranking metrics like Precison@K and Normalized Discounted Cumulative Gain (NDCG).

Both of the approaches fall short of modeling the intrinsic interventionist nature of recommendation, which should not only attempt to model the organic user behavior, but to actually optimally influence it according to a present objective. In other words, in many cases, recommendations are by design actions that have, as a common desired outcome, the change in user behavior, e.g. an increase in a certain user activity such as buying products, watching movies or applying for a mortgage/credit card.

Ideally, the increase in user activity would be measured against the baseline case where no recommendations are being shown. Using a causal vocabulary, we are interested in finding the optimal treatment recommendation policy that maximizes the reward with respect to the control recommendation policy for each user, also known as the Individual Treatment Effect (ITE) (Rubin, 1974). In this context, the control policy constitutes the current state of the system. This can be either a system with no recommendations in place or a baseline recommender system that we want to improve using the currently logged feedback. In the rest of the paper, we will use control and logging policy interchangeably to describe the baseline system.

Currently, this type of approach is prevalent in fields such as personalized medicine, where one tries to infer which treatment would have the biggest positive impact for a particular patient. Similar questions are starting to be asked in the field of Performance Advertising (Johnson et al., 2015), where the problem of measuring the incremental effect of an advertising campaign on the user shopping propensity is very important for proper credit attribution in the context of single and multi-channel advertising (Berman, 2016). In terms of associated rewards, this marks a natural evolution of the field from measuring advertising performance in terms of number of ad displays (CPM - cost per thousand ad displays), to measuring it in terms of ad clicks (CPC), to post-click sales/conversions (CPA) to possibly incremental sales (incremental CPA - value of a sale times the associated individual treatment effect).

In our paper, we introduce a simple modification of standard matrix factorization methods that leverages a small sample of randomized recommendations outcomes in order to create user and products representations. For our method, the associated pairwise distance between user and item pairs is more aligned with the corresponding ITE than in traditional approaches. In Section 4, we show that, because of this, our method can lead to significant lifts in performance over both classical and recent causal inference-based recommendation methods.

1.1. Learning Recommendation Policies Optimized for ITE

In order to introduce our method, we adopt a notation similar to the one introduced in (Swaminathan and Joachims, 2015a) and (Rosenfeld et al., 2016) and extend it to the matrix factorization case. The primary notation used throughout this section is detailed in Table 1.

Recommendation Policy

Consider a recommender system that takes as input a user $u_{i}\in\mathcal{X}$, from the user population $\mathcal{X}$ and outputs as its prediction one of the possible recommendable products $p_{j}\in\mathcal{P}$. The external behavior of the recommender system can be described by the policy of which product to show for each user. We assume a stochastic policy $\pi_{x}$ that associates to each user $u_{i}$ and product $p_{j}$ a probability for the user $u_{i}$ to be exposed to the recommendation of product $p_{j}$:

[TABLE]

For simplicity we assume showing no products is also a valid intervention in $\mathcal{P}$.

Policy Rewards

We define $r_{ij}$ to be the true outcome/reward for recommending product $p_{j}$ to user $u_{i}$. In our case, $r_{ij}$ is a binary outcome, for example click/no click, sale/no sale. We assume that the reward $r_{ij}$ is distributed according to an unknown conditional distribution $r$ depending on $u_{i}$ and $p_{j}$:

[TABLE]

We define $y_{ij}$ to be the observed reward for the pair $i,j$ of user-product according to the logging policy $\pi_{x}$:

[TABLE]

The reward $R^{\pi_{x}}$ associated with a policy $\pi_{x}$ is equal to the sum of the rewards collected across all incoming users by using the associated personalized product exposure probability:

[TABLE]

where the probability of seeing users comes from an unknown distribution $p(\mathcal{X})$: $u_{i}\sim p(\mathcal{X})$ and $R_{ij}$ is the reward associated with the user-product pair $ij$.

The Individual Treatment Effect (ITE) value of a policy for a given user $i$ and a product $j$ for a policy $\pi_{x}$ is defined as the difference between it’s reward and the reward of the control policy $\pi_{c}$:

[TABLE]

In our paper, we are interested in finding the policy $\pi^{*}$ with the highest sum of ITEs:

[TABLE]

where: $ITE^{\pi_{x}}=\sum_{ij}ITE_{ij}^{\pi_{x}}$

Lemma 1

For any control policy $\pi_{c}$, the best incremental policy $\pi^{*}$ is the policy that shows deterministically to each user the product with the highest associated reward.

Proof

Let $\pi^{*}$ be the policy with the highest associated reward:

[TABLE]

We have that $\pi^{*}$ is the policy with the highest ITE w.r.t to any control policy $\pi_{c}$ since $ITE^{\pi^{*}}=R^{*}-R_{c}\geq R^{\setminus*}-R_{c}$.

Let $p_{i}^{*}$ be the product with highest reward for user $u_{i}$:

[TABLE]

and $r_{i}^{*}$ the associated reward.

For any arbitrary policy $\pi_{x}$ we have the following inequality that holds:

[TABLE]

where $\pi_{det}$ is the deterministic policy of showing the best personalized product to each user $u_{i}$:

[TABLE]

Therefore, $\pi^{*}=\pi_{det}$.                                           $\square$

In order to find the optimal policy $\pi^{*}$, we need to find for each user $u_{i}$ the product with the highest personalized reward $r_{i}^{*}$. However, in practice we do not observe directly $r_{ij}$, but $y_{ij}\sim r_{ij}\pi_{x}(p_{j}|u_{i})$. Most of the current approaches use Inverse Propensity Scoring (IPS)-based methods (see (Liang et al., 2016a) and (Schnabel et al., 2016)) to predict the unobserved reward $r_{ij}$:

[TABLE]

The main shortcoming of these approaches is that IPS-based estimators do not handle well big shifts in exposure probability between treatment and control policies (Bottou et al., 2013) (products with low probability under the logging policy $\pi_{c}$ will tend to have higher predicted rewards).

Ideally, for minimum variance, the logging policy $\pi_{c}$ should be one of uniform exposure recommendations (denoted as $\pi^{rand}$), which is impossible in practice due to the resulting low recommendation quality. One potential solution is then to use the biased data from the current $\pi_{c}$ and learn to predict the outcomes under a randomized policy and then rank the products by their predicted outcomes under this transformation. More formally, if we denote by $L(.)$ the loss associated with our policy learning objective, we have that:

[TABLE]

Assuming $\hat{p_{i}}$ is computed as the maximum over predicted rewards we have that:

[TABLE]

where $\hat{y}^{rand}_{ij}$ is the predicted outcome of showing product $p_{j}$ to user $u_{i}$ under randomized exposure and all $\pi^{rand}(p_{j}|u_{i})$ values are equal by the definition of randomized exposure.

Therefore, if we can build a good predictor for $\hat{y}_{ij}^{rand}$, we can avoid the explosion in variance we encounter in the classic IPS approach.

The key difference between domain adaptation and IPS-based methods is that in order to be able to learn a model that performs well on the fully randomized domain, we need a sample from it. That means that at learning time we assume the existence of two training samples: $S_{c}={\{(u_{i},p^{c}_{j},y^{c}_{ij})\}}^{M_{c}}_{i=1}$, that is a very large sample of exposed users with outcomes collected with the control recommendation policy and $S_{t}={\{(u_{i},p^{t}_{j},y^{t}_{ij})\}}^{M_{t}}_{i=1}$, that is a much smaller sample of exposed users with outcomes collected with the fully randomized recommendation policy (for example, collected using an e-greedy exploration strategy (Sutton and Barto, 1998) or from other exploration strategies that can be used to simulate outcomes from $\pi^{rand}$ by rejection sampling). A new class of embedding-based approaches which leverage this setup have been recently proposed in (Johansson et al., 2016; Shalit et al., 2016) and the closely related work in (Rosenfeld et al., 2016).

In our work, we leverage their findings and propose CausalEmbed (CausE), a new embedding method that is able to transfer evidence between a large sample of logged feedback under the control policy $S_{c}$ and a small sample of logged treatment policy feedback $S_{t}$ in order to be able to better predict randomized treatment effects on pairs of users and products. We show that our approach significantly improves recommendation performance on unseen pairs of users and products exposed according to the randomized treatment policy $\pi^{rand}_{t}$.

The key contributions of our work are the following:

• •

We extend the past work (Rosenfeld et al., 2016) on learning from biased control plus randomized treatment and make the connection with the recent work on causal embeddings (Shalit et al., 2016) and propose a new approach entitled CausE, a new matrix factorization algorithm that generalizes both previous approaches.

• •

We apply the newly introduced algorithm on the problem of product recommendation and benchmark it against standard factorization and IPS-based causal methods and analyze various ways of leveraging the exploration sample $S_{t}$ in order to improve the performance of the model on unseen test data from the randomized treatment policy.

To aid the reproducibility of our CausE method, we release the source code111https://github.com/criteo-research/CausE and present results only on public benchmark datasets.

The structure of the paper is the following: In Section 2 we cover previous related work and the relationship with our method. In Section 3 we present the CausE approach. In Section 4 we present the experimental setup and the results on the MovieLens dataset. In Section 5, we summarize our findings and conclude with future directions of research.

2. Related Work

The body of work focusing upon causality is vast and diverse, spanning multiple interconnected disciplines. We concentrate our review on the work measuring causal effects as a counterfactual inference problem.

2.1. Propensity Scoring Methods

The basic idea of propensity scoring methods is to turn the outcomes of an observational study into a pseudo-randomized trial by re-weighting samples, similarly to importance sampling. This is the classical Inverse Propensity Scoring (IPS) (Rosenblum and Rubin, 1983) method of computing an unbiased estimator of the reward of policy $\pi_{t}$ under its own induced distribution from a sample of data collected under policy $\pi_{c}$.

In (Bottou et al., 2013) the authors introduce Clipped Inverse Propensity Scoring, a method which addresses the variance problems that the unbiased IPS estimator is facing when $\pi_{t}$ and $\pi_{c}$ are too different.

Additionally, (Dudík et al., 2011) introduces Double Robust estimation for IPS, as a way to evaluate the value of a new policy on logged data that combines the direct method estimation with the IPS estimation and provides an estimator which is consistent if at least one of the two models is well-specified.

In a series of papers (Swaminathan and Joachims, 2015b, a), the authors propose the Counterfactual Risk Minimization (CRM) principle along with a new learning setup Batch Learning from Bandit Feedback (BLBF) that aims to answer the counterfactual question of: ”How will a system perform in response to an intervention that changes the data distribution?”. As a solution, they propose the Policy Optimizer for Exponential Models (POEM) algorithm, that optimizes a variance-regularized version of the Clipped-IPS estimator. In the context of learning under the BLBF setup, (Lefortier et al., 2016) releases a dataset with logged propensity weights according to a baseline policy $h_{0}$ and benchmarks various propensity weighting methods using the CRM principle.

The most recent work in the BLBF setup introduces BanditNet, an algorithm for training deep neural networks via logged contextual bandit feedback data (Joachims et al., 2018). This approach views the output of a neural network to be analogous with that of a a stochastic policy $\pi_{w}$, where $w$ are the parameters associated with the model. Using BanditNet, $\pi_{w}$ can be trained in the following manner:

[TABLE]

where $\pi_{c}$ is an existing training policy, $\delta_{i}$ is the received loss for $p_{j}$ given $u_{i}$ under $\pi_{c}$ and $\lambda$ is an unconstrained Lagrange multiplier, which must be chosen empirically.

2.2. Causal Inference using Domain Adaptation and Transfer Learning Methods

Causal inference as Domain Adaptation (Johansson et al., 2016; Shalit et al., 2016): The authors frame the problem of causal inference as a domain adaptation problem where we need to learn on the factual domain and predict on the counterfactual domain. By leveraging the results on Domain Adaptation from (Courty et al., 2016), the authors show that the expected ITE estimation error of a representation is bounded by the sum of the supervised generalization error of the representation and the distance between the resulting treated and control distributions in the new representation.

Causal inference as Transfer Learning/Joint Optimization (Rosenfeld et al., 2016): The authors propose a new method that allows for better prediction of the magnitude of a new treatment effect by leveraging not only the A/B Test traffic where the traffic is exposed to the new treatment, but also the pre-A/B Test data that was exposed to the control policy.

2.3. Causal Recommendations

The current work in causal recommendations focuses on the idea of using causal inference as a way to de-bias matrix factorization on existing logged data. In (Liang et al., 2016b) the authors present ExpoMF, a probabilistic approach for collaborative filtering on implicit data. ExpoMF jointly models both users exposure to an item, and their resulting click decision, resulting in a model which naturally down-weights the expected, but ultimately un-clicked items. The model is also able to consider additional covariates which may impact the exposure, for example a user’s location when recommending restaurants. In (Liang et al., 2016a) the authors follow-up with a second version of the model for rating matrices and with explicit modeling of the exposure model (they propose two exposure models, one of global popularity and one that personalizes the exposure for each user). Once the exposure model is estimated the preference model is fit with weighted click data, where each click (or skip) is weighted by the inverse probability of exposure (coming from the exposure model). As a result, the observational click data is weighted as though it came from an ”experiment” where users are randomly shown items. This leads to good performance on the domain-adaptation task where the model is estimated on a different distribution from the training one. A very similar model was proposed in (Schnabel et al., 2016).

3. Proposed Approach

As discussed in Section 1, we are interested in building a good predictor for recommendation outcomes under random exposure for all the user-product pairs, which we denote as $\hat{y}_{ij}^{rand}$. In our learning setup, we assume that we have access to a large sample $S_{c}$ from the logging policy $\pi_{c}$ and a small sample $S_{t}$ from the randomized treatment policy $\pi_{t}^{rand}$. To this end, we propose a multi-task objective that jointly factorizes the matrix of observations $y^{c}_{ij}\in S_{c}$ and the matrix of observations $y^{t}_{ij}\in S_{t}$.

3.1. Multi-Task Objective For Exposure Policies That Vary Treatments For A Given Set Of Fixed Users

For simplicity we assume that the same set of users are exposed differently in control/treatment and the exposure change between treatment and control is explained by the difference in product representations in the two policies. In this case, the users have a fixed representation (either as a set of user features or as a user embedding vector). This is the evaluation setup proposed in (Liang et al., 2016a) and the equivalent evaluation of the work in (Schnabel et al., 2016), where the test sample is produced according to $\pi^{rand}_{t}$.

The case of predicting outcomes under changing treatment exposures for the same set of users is also addressed by the authors in (Rosenfeld et al., 2016). They assume a user-independent exposure policy and derive a multi-task objective that explains jointly the small sample $S_{t}$ and the big sample $S_{c}$ and regularizes the difference in the two resulting product representations. In order to motivate this objective, the authors assume that both the expected factual control and treatment rewards can be approximated as linear predictors over the fixed user representations $u_{i}$:

[TABLE]

where $\theta^{c}_{j},\theta^{t}_{j}$ are the control/treatment vectorial representations of product $j$ and $<x,y>$ denotes the inner product between the vectors $x$ and $y$.

As a result and using the logic of Equation 3, we can approximate the ITE of a user-product pair $i,j$ as the difference between the two:

[TABLE]

We define the first part of our joint prediction objective as the supervised predictor for $y^{t}_{ij}$, trained on the limited sample $S_{t}$. We denote the associated loss term by $l^{t}_{ij}$:

[TABLE]

where: $L$ is an arbitrary loss function and and $\Omega(.)$ is a regularization term over the weights of the model. Switching to matrix notation, we define the associated objective by $L_{t}$:

[TABLE]

where: $\Theta_{t}$ is the parameter matrix of treatment product representations, $U$ is the fixed matrix of the user representations, $Y_{t}$ is the observed rewards matrix, $L(.)$ is an arbitrary loss function and $\Omega(.)$ is a regularization term over the weights of the model.

Then, if we want to leverage the ample control data, we can use our treatment product representations by subtracting $\theta^{\Delta}_{j}$ since $<\theta^{c}_{j},u_{i}>=<\theta^{t}_{j},u_{i}>-<\theta^{\Delta}_{j},u_{i}>$. We denote the associated objective by $l^{c}_{ij}$:

[TABLE]

If instead of parametrizing it in terms of treatment embeddings $\theta^{t}_{j}$ and a set of difference vectors $\theta^{\Delta}_{j}$, we choose to express the loss in terms of the new control embedding $\theta^{c}_{j}$, and switching to matrix notation, we have that:

[TABLE]

By putting the two tasks together (eq. 15 and 17), we have the equation of the joint tasks loss $L^{prod}_{CausE}$:

[TABLE]

We can similarly define $L^{user}_{CausE}$, where the user embeddings are learnt over fixed product representations. This type of modelization is relevant for the cases where we know that the treatment policy differs from the control policy in terms of the choice of users.

3.1.1. Multi-task Objective For Exposure Policies that Vary Both Treatments And Users

In this case, the change in exposure between treatment and control is explained both by the different choice in exposed users and products shown. To this end, we modify our objective function to allow for the user representations to change, and using similar reasoning to eq (18) we reach $L_{CausE}$, the final loss function for our method:

[TABLE]

We can recover the previous loss as a special case by setting the regularization parameter on the difference between treatment and control representations for users to infinity.

3.1.2. Optimization

We optimize $L_{CausE}$ using Stochastic Gradient Descent (SGD) with momentum (Sutskever et al., 2013) and a linearly decaying learning rate. For our experiments we optimize $L^{prod}_{CausE}$ since our treatment policy depends only on products and not on users. Algorithm 1 details the general CausE method.

4. Experiments

The experimental section is organized as follows. Firstly, we describe the evaluation setup, namely, the evaluation task, success metrics, the baselines. and the dataset preprocessing protocol. In the second part, we report results of our experiments on both the MovieLens10M and Netflix datasets.

4.1. Setup

4.1.1. Task: Estimating Treatment Rewards $\mathbf{y^{t}_{ij}}$

We compare all of the methods on the task of predicting the outcomes $y^{t}_{ij}$ under treatment policy $\pi_{t}$, where all of the methods have available at training time a large sample of observed recommendations outcomes from $\pi_{c}$ and a small sample from $\pi_{t}$ (to simulate the large control plus small treatment info coming from exploration or an A/B test). Since the resulting setup is basically a classical conversion-rate prediction problem, we will use the associated metrics, namely Mean-Squared Error (MSE) and Negative Log-Likelihood (NLL). For our final numbers we report their lift (eq 20) over the performance of the average predictor $AvgCR$ (e.g. the empirical conversion rate on the test dataset), to ensure that the performance of the predictors evaluated on the test distribution is better than a trivial predictor that has access to the test distribution:

[TABLE]

where $metric$ is either MSE or NLL and $x$ is the approach being measured.

In addition, as we are concerned with implicit feedback data which is inherently binary, we also report the area under the receiver operating characteristic curve (AUC) to assess the quality of the predictions made by the various approaches.

4.1.2. Baselines

To demonstrate the effectiveness of our proposed approaches, we compare with the following baselines:

**Matrix Factorization Baselines:

** **Bayesian Personalized Ranking (BPR)

** To compare our approach against a ranking based method, we use Bayesian Personalized Ranking (BPR) for matrix factorization on implicit feedback data (Rendle et al., 2009). In this method, we directly learn user and product representations via matrix factorization, optimized using the BPR-OPT criteria and trained using LearnBPR (Rendle et al., 2009).

**Supervised-Prod2Vec (SP2V)

** As a second factorization baseline we will use a Factorization Machine-like method (Rendle, 2010) that approximates $y_{ij}$ as a sigmoid over a linear transform of the inner-product between the user and product representations:

[TABLE]

where: $\sigma$ is the sigmoid function, $\alpha$ is a scaling factor for the user-product similarity score, $b$ is the global bias, $b_{i}$ and $b_{j}$ are user and product bias terms. This method can also be seen as an extension of the Prod2Vec algorithm (Grbovic et al., 2015) for factorizing non-symmetric matrices in the presence of supervised feedback (e.g. negatives are not randomly sampled as they are provided within the data). For this reason, we will denote our supervised baseline as Supervised-Prod2Vec (SP2V).

**Causal Inference Baselines:

** **Weighted-SupervisedP2V (WSP2V)

** To test the performance of propensity-based causal inference, we employ the SP2V algorithm on propensity-weighted data, which we denote in our experiments as Weighted-SP2V (WSP2V). The method is similar to the Propensity-Scored Matrix Factorization (PMF) from (Schnabel et al., 2016) but with cross-entropy reconstruction loss instead of MSE/MAE. An equivalent method was tested in (Liang et al., 2016a) where the authors argue that their best performing model, ExpoMF trained causally (CAU) with conditional prediction (Cond), is equivalent to PMF.

**BanditNet (BN)

** To utilize BanditNet as a baseline, we use SP2V as our target policy $\pi_{w}$. For the existing policy $\pi_{c}$, we model the behavior of the recommendation system as a popularity-based solution, described by the marginal probability of each product in the training data. In our experiments, in order to keep the magnitude of the resulting factor $\frac{1}{\pi_{c}(p_{j})}$ within a reasonable magnitude we replace it with the $\frac{\pi^{rand}(p_{j})}{\pi_{c}(p_{j})}$ and cap it to maximum 100. Under this setup, BanditNet with $\lambda=0$ has a similar formulation with WSP2V. As in the original paper (Joachims et al., 2018), we train BanditNet using SGD with momentum (Sutskever et al., 2013).

4.1.3. Leveraging The Exploration Sample $\mathbf{S_{t}}$

In order to better understand the leverage we obtain by transferring information from the limited treatment policy sample, we define four possible setups of incorporating the exploration data:

• •

No adaptation (no) - the algorithm does not take into account the exploration data, it is trained only on the $S_{c}$ sample.

• •

Blended adaptation (blend) - the algorithm is trained on the union of the $S_{c}$ and $S_{t}$ samples.

• •

Test-only adaptation (test) - the algorithm is trained only on the $S_{t}$ samples.

• •

Average test adaptation (avg) - the algorithm constructs an average treatment product by pooling all of the points from the $S_{t}$ sample into a single vector (it applies only to CausE).

• •

Product-level adaptation (prod) - the algorithm constructs a separate treatment embedding for each product based on the $S_{t}$ sample (it applies only to CausE). For the final prediction we can use either the control (denoted by CausE-prod-C) or the treatment product representation (denoted by CausE-prod-T).

In the following sub-section we detail our methodology for creating the evaluation datasets.

4.1.4. Datasets: MovieLens10M and Netflix

The datasets used for evaluation are the MovieLens10M (Harper and Konstan, 2016) (with 71567 unique users and 10677 unique products) and Netflix (Bennett et al., 2007) (with 480189 unique users and 17770 unique products) recommendation datasets. We also use the MovieLens100K dataset (Harper and Konstan, 2016) to explore how changes in the quantity of test set injected into the training set affect performance. In order to be able to simulate causal effects under uniform exposure, all datasets are transformed using the standard protocol introduced by Liang et al.(Liang et al., 2016a): First we binarize the ratings $y_{ij}$ by setting all observed five-star ratings to one (click) and everything else to zero (view only). We then create two datasets: regular (REG) and skewed (SKEW), each one with 70/10/20 train/validation/test event splits.

To create the regular dataset, for each user we split the products in train/validation/test, similarly with the normal supervised evaluation setup. Whilst in the creation of the skewed dataset, we first generate the test dataset where we expose as uniformly as possible each user to each product. This simulates a random test recommendation policy $\pi_{t}^{rand}$, which the authors argue is the best policy to collect unbiased feedback for training a recommendation model. In order to achieve this we estimate the popularity of each product and find the least popular product. For all other products we compute how much more common it is than the least popular product. This can then used to compute an acceptance probability e.g. if a product is 100 times more popular than the least popular product we accept these products in the test set with 1% probability.

In the original protocol, the skewed training and validation sets partition the rest of the data to obtain the final 70/10/20 split. This setup is sufficient to be able to compute the propensity weights between the training policy and the test policy, e.g. $\frac{\pi_{t}(p_{j}|u_{i})}{\pi_{c}(p_{j}|u_{i})}$ that will be used by WP2V. However, for CausE we need at training time an explicit sample from $\pi_{t}$ and not only the propensity ratio. Because of this, we increase the size of the uniform test to 30%, 20% used for final testing and 10% to be injected in the training data.

To recap, the final skewed dataset for which we report the results in Table 2 comprises three parts: 70% train (60% from $\pi_{c}$ and 10% from $\pi_{t}$), 10% validation (all from $\pi_{c}$), 20% test (all from $\pi_{t}$). To limit the number of test products that are not available in training, due to fully sampling them for skewed testing, we cap the maximum probability for a product event to be moved in the test split to 0.9. Additionally, to help limit overfitting for WP2V, we cap the maximum propensity weight to 10.

4.2. Results

Table 2 shows the results for all approaches evaluated on the binarized MovieLens10M and Netflix datasets.

We use the standard cross-entropy loss as the loss function $L$. In terms of implementation, all the baseline and CausE approaches tested have been coded in Tensorflow and were optimised using SGD with momentum and a linearly decaying learning rate.

For our choice of regularization functions, similarly with the previous work of (Shalit et al., 2016) and (Rosenfeld et al., 2016), we experimented with both $L_{1}$ and $L_{2}$ for penalizing the embedding matrices and the discrepancy between the two product representations associated with the $S_{c}$ and $S_{t}$ distributions. We report our best results which we reached with the configuration $\Omega_{t,c}(.)=L_{2}$ and $\Omega_{dist}(.)=L_{1}$.

As one would expect, in the case of the REG dataset, where the training and test distributions are constant, all methods demonstrated similar performance. For the sake of brevity we do not present the REG results in Table 2, but for the MovieLens10M dataset all approaches achieved an MSE lift of $+8\%(\pm 0.02)$.

On the SKEW datasets, where the test distribution is very different from the training distribution, the standard matrix factorization and supervised methods (SP2V and BPR) perform badly, with the method having access to the $S_{t}$ sample during training time showing a marginal increase in performance (SP2V-blend). SP2V-test is the only method to be trained exclusively upon data sampled from the test distribution, but as expected, it cannot perform as well as other methods due to sparsity issues arising from its comparatively smaller size.

As expected, the competing causal approaches lead to better results: the IPS-weighted methods (WSP2V-no/blend) can take advantage of the propensity information and are able to outperform the standard supervised methods and manage to exhibit some domain adaptation from $S_{c}$ to $S_{t}$ on both datasets. BanditNet (BN-blend) outperforms the IPS-weighted approaches and demonstrates performance on the test distribution that is the closest to our proposed method.

Finally, our proposed CausE method significantly outperforms all baselines across both datasets, demonstrating that it has a better capacity to leverage the small test distribution sample $S_{t}$. We observe that, out of the three CausE variants, CausE-prod-C, the variant that is using the regularized control matrix, clearly out-performs the others. We believe that this is due to the fact that fitting the larger training sample $S_{c}$ allows to model correctly the personalized user responses, while controlling the deviation from the target task via the discrepancy regularizer.

In terms of the impact of the size of the $S_{t}$ sample on performance, figure 1 shows that CausE is able to leverage the extra data from the test distribution much better than the other methods, giving it the best overall lift.

5. Conclusions

In this paper, we have introduced a novel method for factorizing matrices of user implicit feedback that optimizes for causal recommendation outcomes. We show that the objective of optimizing for causal recommendations is equivalent with factorizing a matrix of user responses collected under uniform exposure to item recommendations and propose CausE, a domain adaptation-inspired method that learns from a large sample of biased exposure feedback data $S_{c}$ and a small sample of unbiased exposure feedback data $S_{t}$. The resulting method is a simple extension to current matrix factorization algorithms that adds a regularizer term on the discrepancy between the item vectors used to fit the biased sample $S_{c}$ and their counter-part representations that fit the uniform exposure sample $S_{t}$.

Since, unlike other competing causal models, we are able to leverage the large sample of biased feedback data, we show that we can clearly outperform both classical matrix factorization methods, in addition to recent causal approaches such as IPS-weighted factorization methods and BanditNet.

We believe this to be promising for industrial applications for two main reasons: Firstly, in real-world scenarios, most of the current user feedback data is collected under a biased recommendation system which fits perfectly in our approach. Secondly, most of real-world recommender systems are based on matrix factorization, which would require little modifications in order to be able to experiment with our proposed approach.

As future work, we plan to extend this approach to user sequence modeling and leverage both organic user activity sequences and influenced user activity sequences in order to make sequential product recommendation policies more incremental.

Acknowledgements

The authors would like to thank David Rohde, Mike Gartrell and Jeremie Mary for their insightful feedback and comments during the preparation of this manuscript.