Cross-Domain Collaborative Filtering via Translation-based Learning

TL;DR

This paper introduces CDT, a translation-based cross-domain collaborative filtering model that learns user preferences across domains using embedding translations, and extends it with deep learning for improved non-linear preference modeling, outperforming existing methods.

Contribution

The paper proposes a novel translation-based framework for cross-domain collaborative filtering and extends it with deep learning to better capture complex user preferences.

Findings

CDT outperforms state-of-the-art cross-domain recommendation methods.

The translation-based approach effectively models high-order feature interactions.

DeepCDT further improves recommendation accuracy by capturing non-linear preferences.

Abstract

With the proliferation of social media platforms and e-commerce sites, several cross-domain collaborative filtering strategies have been recently introduced to transfer the knowledge of user preferences across domains. The main challenge of cross-domain recommendation is to weigh and learn users' different behaviors in multiple domains. In this paper, we propose a Cross-Domain collaborative filtering model following a Translation-based strategy, namely CDT. In our model, we learn the embedding space with translation vectors and capture high-order feature interactions in users' multiple preferences across domains. In doing so, we efficiently compute the transitivity between feature latent embeddings, that is if feature pairs have high interaction weights in the latent space, then feature embeddings with no observed interactions across the domains will be closely related as well. We formulate our objective function as a ranking problem in factorization machines and learn the model's parameters via gradient descent. In addition, to better capture the non-linearity in user preferences across domains we extend the proposed CDT model by using a deep learning strategy, namely DeepCDT. Our experiments on six publicly available cross-domain tasks demonstrate the effectiveness of the proposed models, outperforming other state-of-the-art cross-domain strategies.

Tables

Table 1. TABLE I : The six cross-domain recommendation tasks.

Domain	Users	Items	Ratings
Electronics	18,649	3,975	23,009
Kitchen	16,114	5,511	19,856
Toys	9,924	3,451	13,147
DVD	49,151	14,608	124,438
Music	69,409	24,159	174,180
Video	11,569	5,223	36,180

Table 2. TABLE II : Effect on recall. Bold values denote the best scores, using the paired t-test ( p < 𝑝 absent p< 0.05). The underlined values denote the second best method.

	Electronics	Kitchen	Toys	DVD	Music	Video
FMs	.183	.124	.138	.462	.431	.601
CML	.201	.139	.142	.488	.429	.629
FNN	.212	.136	.146	.502	.447	.613
CBT	.221	.142	.157	.519	.451	.653
CLFM	.239	.151	.166	.534	.458	.672
CDCF	.234	.155	.171	.568	.462	.684
ScoNet	.248	.173	.176	.605	.493	.697
CDT	.241	.178	.185	.583	.488	.706
DeepCDT	.265	.194	.204	.634	.524	.713

Equations

$$\begin{array}[]{c}\min\limits_{Y^{(pt)}}||O^{(pt)}-C^{(p)}Y^{(pt)}{C^{(t)}}^{\top}||_{F}^{2}+\lambda||Y^{(pt)}||_{2,1}\\ \text{subject to ${Y^{(pt)}}^{\top}Y^{(pt)}=I$, $Y^{(pt)}\geq 0$}\end{array}$$

$$Q^{(pt)}(h,u)=\frac{\sum\limits_{k\in\mathcal{I}^{(p)}_{h}}Y^{(pt)}\big{(}C^{(p)}(k),C^{(t)}(u)\big{)}}{\big{|}\mathcal{I}^{(p)}_{h}\big{|}}$$

$$x^{(t)}=\big{\{}\underbrace{(u,t),(i,t),}_{\text{target knowledge}}\underbrace{z^{(1)}(u),\ldots,z^{(d-1)}(u)}_{\text{source knowledge}}\big{\}}$$

$$\hat{y}(x^{(t)})=w_0+\sum _{i=1}^l{w}_i{x}_i^{(t)}+\sum _{h=i+1}^l{d}^2(v_i+v_i^{\prime },v_h)x_i^{(t)}{x}_h^{(t)}$$

$$\hat{\Theta }=\arg \underset{\Theta }{\max }=\prod _{u\in {\mathcal{U}}^{(t)}}\prod _{i^{\prime }\in {\mathcal{I}}_u}\prod _{j\in {\mathcal{I}}^{\prime (t)}}Pr(i^{\prime }{>}_{u}j|\Theta )Pr(\Theta )$$

$$\hat{\Theta}=\arg\max_{\Theta}=\prod_{u\in{\mathcal{U}^{(t)}}}\prod_{i^{\prime}\in\mathcal{I}_{u}}\prod_{j\in\mathcal{I}^{\prime(t)}}\ln\sigma\big{(}\hat{y}(x^{(t)}_{u,i^{\prime}})-\hat{y}(x^{(t)}_{u,j})\big{)}-\Omega(\Theta)$$

Full text

I Introduction

The collaborative filtering strategy has been widely followed in recommendation systems, where users with similar preferences tend to get similar recommendations. User preferences are expressed explicitly in the form of ratings or implicitly in the form of number of views, clicks, purchases, and so on. Representative collaborative filtering strategies are latent models such as Matrix Factorization and Factorization Machines (FMs) [1], which factorize the data matrix with user preferences in a single domain (e.g., music or video), to reveal the latent associations between users and items. However, data sparsity and cold-start problems degrade the recommendation accuracy, as there are only a few preferences on which to base the recommendations in a single domain. With the advent of social media platforms and e-commerce systems, such as Amazon and Netflix, users express their preferences in multiple domains. For example, in Amazon users can rate items from different domains, such as books and retail products, or users express their opinion on different social media platforms, such as Facebook and Twitter. In the effort to overcome the data sparsity and cold-start problems, several cross-domain recommendation strategies have been proposed, which exploit the additional information of user preferences in multiple auxiliary/source domains to leverage the recommendation accuracy in a target domain [2, 3, 4, 5, 6]. However a pressing challenge resides on how to transfer the knowledge of user preferences from different domains, by also weighting the importance of users’ different behaviors accordingly.

In cross-domain recommendation, the source domains can be categorized based on users’ and items’ overlaps, that is, full-overlap, and partial or non user/item overlap between the domains. In this study, we focus on partial users’ overlaps between the target and the source domains, as it reflects on the real-world setting [7]. Cross-domain recommendation algorithms differ in how the knowledge of user preferences from the source domains is exploited, when generating the recommendations in the target domain. For example, Li et al. [3] calculate user and item clusters for each domain, and then encode the cluster-based patterns in a shared codebook; finally, the knowledge of user preferences is transferred across domains through the shared codebook. Gao et al. [8] present a Cluster-based Latent Factor Model which uses joint nonnegative tri-factorization to construct a latent space to represent the rating patterns of user clusters on the item clusters from each domain, and then generates the cross-domain recommendations based on a subspace learning strategy. Cross-Domain collaborative filtering with FMs, presented in [4], is a state-of-the-art cross-domain recommendation. It is a context-aware approach which applies factorization on the merged domains, aligned by the shared users, where the source domains are used as context. Hu et al. [7] jointly learn neural networks to generate cross-domain recommendations based on stich units, introducing a shared auxiliary matrix to couple two hidden layers when training the networks in parallel. However, these cross-domain recommendation strategies do not pay attention to users’ complex behaviors across domains, where we have to weigh and learn high-order feature interactions of users’ preferences while transferring the knowledge of users’ selections from multiple domains.

Recently, metric/translation-based models have been shown to be effective in collaborative filtering [9]. Essentially the idea behind is to learn an embedding and translation space for each feature dimension, replacing the inner product of latent models with the squared Euclidean distance to measure the interaction strength between features. By learning a latent item embedding space with translation vectors, such models can better capture the transitivity between feature latent embeddings also known as the similarity propagation process, that is if feature pairs have high interaction weights in the latent space, then feature embeddings with no observed interactions will be closely related as well. In a similar spirit, recent studies demonstrate the effectiveness of translation vectors in sequential recommendation [10, 11]. Nonetheless, these models produce recommendations in a single-domain, and omit users’ various and complex feature interactions across domains.

To overcome the shortcomings of existing strategies, we propose a Cross-Domain Translation-based learning model, namely CDT, making the following contributions: We first compute the cluster-based users’ similarities across domains, and then we propose a metric/translation-based learning strategy to compute the embedding space with translation vectors and capture high-order feature interactions when optimizing user preferences across domains. We formulate our objective function as a ranking problem in FMs and learn the model’s parameters via gradient descent. In addition, to efficiently learn the non-linearity in user preferences across domains we extend the proposed CDT model by using a deep learning strategy, namely DeepCDT. In our experiments on six cross-domain tasks, we show that our models outperform other single-domain and cross-domain strategies.

II The Proposed Model

In our setting we assume that we have $d$ different domains, where $n_{p}$ and $m_{p}$ are the numbers of users and items in the $p$-th domain, respectively. In matrix $R^{(p)}$, we store the user preferences on items, in the form of explicit feedback e.g., ratings or in the form of implicit feedback e.g., number of views, clicks, and so on. In this study we consider users’ partial overlaps across the domains. We define a users’ overlapping matrix $O^{(pt)}\in\mathbb{R}^{n_{p}\times n_{t}}$ between a source domain $p$ and the target domain $t$. For each cell holds $O^{(pt)}(k,u)$= 1, if users $k$ and $u$ are the same user in domains $p$ and $t$, and 0 otherwise. The goal of the proposed CDT model is to generate personalized recommendations in the target domain $t$, while transferring and weighting users’ different preferences from the $d-1$ source domains. The proposed CDT model consists of a cross-domain co-clustering strategy to compute the cluster-based similarities among different users across domains, and our translation-based strategy to generate cross-domain recommendations.

II-A Cross-domain Co-Clustering

In each domain we first compute the user cluster assignment matrices $C^{(p)}\in\mathbb{R}^{n_{p}\times c_{p}}$ and $C^{(t)}\in\mathbb{R}^{n_{t}\times c_{t}}$, where $c_{p}$ and $c_{t}$ are the number of user clusters in the $p$-th domain and the target domain $t$, respectively. In our implementation we compute matrices $C^{(p)}$, with $p=1,\ldots,d-1$, and matrix $C^{(t)}$ based on the graph Laplacian method [12]. To calculate the cluster-based similarities of users in matrix $Y^{(pt)}\in\mathbb{R}^{c_{p}\times c_{t}}$ between domains $p$ and $t$, we follow a co-clustering strategy for each source domain $p$ and the target domain $t$, trying to minimize the following objective function:

[TABLE]

$||\cdot||_{2,1}$ denotes the $L_{2,1}$ norm of a matrix forcing $Y^{(pt)}$ to be sparse, reflecting on the real-world scenario, where users’ overlaps are usually sparse [13]. Then, to compute the common user’s $u$ behavior in domains $p$ and $t$ we calculate each cell of matrix $Q^{(pt)}\in\mathbb{R}^{m_{p}\times n_{t}}$ based on the entries of the cluster-based user similarities in $Y^{(pt)}$ as follows:

[TABLE]

where $\mathcal{I}^{(p)}_{h}$ is the set of users $k=1,\ldots,n_{p}$ that have interacted with item $h=1,\ldots,m_{p}$ in the $p$-th domain, and $(k,h)\in R^{(p)}$.

II-B Cross-domain Translation-based Learning

II-B1 Feature Vector

For the target domain $t$ we represent each user-item interaction $(u,i)\in R^{(t)}$ as a feature vector $f\in\mathbb{R}^{n_{t}+m_{t}}$, with $u=1,\dots,n_{t}$ and $i=1,\ldots,m_{t}$. Using a lookup operation, the feature vector is expressed by its sparse representation as follows: f^{(t)}=\big{\{}(u,t),(i,t)\big{\}}. For each source domain $p=1,\dots,d-1$, we extend $f^{(t)}$ with a vector z^{(p)}(u)=\big{\{}h,Q^{(pt)}(h,u)\big{\}}, with $h=1,\ldots,m_{p}$ and $(u,h)\in R^{(p)}$. When computing the vector $z^{(p)}(u)$, matrix $Q^{(pt)}$ weighs the ratings on $h$ of user $u$ in the $p$-th domain. By augmenting the feature vector $f^{(t)}$ with the $d-1$ vectors $z^{(p)}(u)$ of the source domains, its sparse representation becomes an $l$-dimensional vector as follows:

[TABLE]

Notice that vector $x^{(t)}$ contains the user-item interactions in domain $t$, as well as the interactions in $p$, weighted by $Q^{(pt)}$ based on Eq. (2).

II-B2 Embedding and Translation Vectors

For each dimension of $x^{(t)}$ we try to learn the embedding vectors $v_{i}\in\mathbb{R}^{q}$, $v_{h}\in\mathbb{R}^{q}$ and a translation vector $v^{\prime}_{i}\in\mathbb{R}^{q}$, where $q$ is the number of latent dimensions, $i\in\mathcal{I}^{(t)}$ is an item that belongs to the item set that a user $u$ has rated in the target domain $t$ and $h\in\mathcal{I}^{(p)}$ in the source domain $p$. To capture the user’s $u$ transition from item $i$ in domain $t$ to item $h$ in domain $p$, we have to learn a translation vector as follows: $v_{i}+v^{\prime}_{i}\approx v_{h}$. This means that $v_{h}$ should be a nearest neighbor of $v_{i}+v^{\prime}_{i}$ in the $q$-th dimensional latent space according to a metric-based metric e.g., the squared Euclidean distance $d^{2}(v_{i}+v^{\prime}_{i},v_{h})$. Notice that the distance $d^{2}(v_{i}+v^{\prime}_{i},v_{j})$ replaces the inner product term of conventional latent models, following the metric/translation-based learning strategy [9]. In doing so, the proposed CDT model can effectively capture the transitive property between feature embeddings of domains $t$ and $p$ in the the latent space. As a consequence, in our cross-domain model if feature pairs have high interaction weights in the latent space, then feature embeddings with no observed interactions across the domains will be closely related as well. Based on the collaborative filtering strategy of FMs [1], we can formulate the model Equation of CDT as follows:

[TABLE]

$w_{0}$ is the global bias, $w_{i}$ is the linear term for feature $x^{(t)}_{i}$. Accordingly, $v_{i}$ and $v^{\prime}_{i}$ are the embedding and translation vectors for feature $x^{(t)}_{i}$.

II-B3 Objective Function

Following the Bayesian Personalized Ranking criterion [14], we try to rank higher the observed items $i\in\mathcal{I}^{(t)}$ than the unobserved ones $j\in\mathcal{I}^{\prime(t)}$ for a user $u\in\mathcal{U}^{(t)}$ in the target domain $t$. In addition, in our cross-domain setting we have to rank higher the observed items $h\in\mathcal{I}^{(p)}$ in a source domain $p$ than the unobserved ones $j\in\mathcal{I}^{\prime(t)}$ in the target domain $t$. To describe these relations we first consider a global item set $\mathcal{I}_{u}$ that user $u$ has rated in all the $d$ domains. Then, we define the partial relation $i^{\prime}>_{u}j$, with $i^{\prime}\in\mathcal{I}_{u}$. In our model, we formulate the following objective function:

[TABLE]

with $\Theta$ being the set of the model parameters. According to the model Equation of CDT in Eq. (4), the objective function of Eq. (5) is formulated as follows:

[TABLE]

where $\sigma$ the sigmoid function and $\Omega(\Theta)$ is the $L_{2}$-norm regularization term on the model’s parameters. To compute the model’s parameters we transform the objective function of Eq (6) as a minimization problem and optimize it via gradient descent using negative sampling, that is randomly selecting unobserved items in the target domain $t$. In our implementation, we used five negative samples for each positive/observed sample in the target domain $t$, as we found out that for larger numbers of negative samples the computational cost of the model learning did not pay off in terms of recommendation accuracy

II-B4 DeepCDT

As the user-item interactions are non-linearly associated across the domains, we implemented a variant of the proposed CDT model, namely DeepCDT. Using the model Equation of CDT in Eq. (4) as the bottom layer of a deep neural network with $e$ hidden layers, we learn the deep representations of the translated-based features of Section II-B2, in a similar way as the single-domain deep learning strategy of FMs in [15]. In our implementation we used Tensorflow111https://www.tensorflow.org and fix the number of hidden layers $e$=5. We computed the model’s parameters, that is the weight matrices of the neural network via backpropagation with stochastic gradient descent, trying to optimize the objective function in Eq. (6). We employed mini-batch Adam which adapts the learning rate for each parameter by performing smaller updates for frequent and larger updates for infrequent parameters. We set the batch size of mini-batch Adam to 512 with a learning rate of 1e-4. Also, we varied the number of latent dimensions $q$ from 10 to 100 by a step of 10, using a grid selection strategy and we kept the latent dimensions fixed based on cross-validation.

III Experiments

III-A Setup

III-A1 Cross-domain Tasks

Our experiments were performed on six cross-domain tasks from the Amazon dataset [16]. The items are grouped in categories/domains, and we evaluate the performance of our model on the six largest domains. The main characteristics of the evaluation data are presented in Table I.

III-A2 Evaluation Protocol

In each out of the six cross-domain recommendation tasks, the goal is to generate recommendations for a target domain, while the remaining five domains are considered as source domains. We trained the examined models on the 50% of the target domain and all the ratings of the source domains as training set. We used 10% of the ratings in the target domain as cross-validation set to tune the models’ parameters and evaluate the examined models on the remaining test ratings. To remove user rating bias from our results, we considered an item as relevant if a user has rated it above her average ratings and irrelevant otherwise [17]. We measured the quality of the top-$n$ recommendations in terms of the ranking-based metrics recall and Normalized Discounted Cumulative Gain (NDCG@n). Recall is the ratio of the relevant items in the top-$n$ ranked list over all the relevant items for each user. NDCG measures the ranking of the relevant items in the top-$n$ list. For each user the Discounted Cumulative Gain (DCG) is defined as: $DCG@n=\sum_{j=1}^{n}{\frac{2^{rel_{j}}-1}{\log_{2}{j+1}}}$, where $rel_{j}$ represents the relevance score of item $j$, that is binary in our case, i.e., relevant or irrelevant. NDCG is the ratio of DCG/iDCG, where iDCG is the ideal DCG value given the ratings in the test set. We fixed the number of recommendations to $n$=10. We repeated our experiments five times and averaged recall and NDCG over the five runs.

III-A3 Compared Methods

We compare the proposed models CDT and DeepCDT with the single-domain strategies FMs [1], CML [9] and FNN [15], and the cross-domain strategies CBT [3], CLFM [8], CDCF [4] and ScoNet [7]. The parameters of the examined methods have been determined via cross validation and in our experiments we report the best results.

III-B Performance Evaluation

Tables II presents the experimental results in terms of recall. In addition, Figure 1 shows the effect on NDCG for the cross-domain models in the largest domain “Music”, by varying the training set size. The cross-domain models CBT, CLFM, CDCF, ScoNet, CDT and DeepCDT significantly outperform the single-domains models of FMs CML and FNN, by exploiting users’ preferences in the source domains when generating recommendations, thus reducing the data sparsity in the target domain. The proposed CDT model clearly beats the baseline cross-domain strategies CBT and CLFM, and the cross-domain strategy CDCF with FMs, demonstrating that our translation-based strategy plays a crucial role in boosting the cross-domain recommendation accuracy. To further verify the importance of our translation-based strategy, we observe that DeepCDT achieves higher performance than FNN. Notice that instead of feeding the neural network with the features of FMs from a single-domain as FNN [15], in our cross-domain setting we feed the neural network with the translated-based features, learned in Section II-B2, which explains DeepCDT high recommendation accuracy. ScoNet and CDT have comparable performance because ScoNet captures the non-linear associations between user preferences across the domains, while CDT only models the high-order feature interactions in users’ behaviors between the domains. Using the paired-t test we found out that DeepCDT is superior over all the competitive approaches for $p<$0.05, by taking into account both the non-linearity and high-order feature interactions in users’ behaviors across multiple domains.

IV Conclusions

We presented CDT, a translation-based model for generating cross-domain recommendations. The key idea of our CDT model is to transfer the knowledge of users’ preferences across domains, while computing the translation-based feature interactions, and consequently capturing complex high-order feature interactions in different domains. In addition, we introduce a deep learning variant of our model, namely DeepCDT to efficiently compute the non-linear associations of features in users’ behaviors across the domains. Our experiments showed that the proposed approaches significantly outperform other baseline methods, proving the importance of our cross-domain translation-based learning strategy.

As future work, we plan to extend the proposed CDT and DeepCDT models for sequential recommendations in cross-domain tasks. Generating sequential recommendations is a challening task, where the goal is to predict the next item that a user will select. Although there are many single-domain strategies for sequential recommendation, such as the studies reported in [10, 11], the case of cross-domain reflects better on the real-world scenario, where not only we have to capture users’ sequential behaviors, but transfer this knowledge across different domains. In addition, we plan to study the influence of the proposed method on the link prediction task across multiple platforms [18], as well as on modeling users’ preference dynamics [19].