Personalized Neural Embeddings for Collaborative Filtering with Text

TL;DR

This paper introduces PNE, a neural embedding framework that combines user-item interactions and item text to improve recommendation accuracy, addressing data sparsity in collaborative filtering.

Contribution

The paper presents a novel joint embedding approach that integrates textual information with interaction data for personalized recommendations.

Findings

PNE outperforms four state-of-the-art baselines on real-world datasets.

PNE effectively learns meaningful word embeddings.

The model improves recommendation accuracy by leveraging text and interaction data.

Abstract

Collaborative filtering (CF) is a core technique for recommender systems. Traditional CF approaches exploit user-item relations (e.g., clicks, likes, and views) only and hence they suffer from the data sparsity issue. Items are usually associated with unstructured text such as article abstracts and product reviews. We develop a Personalized Neural Embedding (PNE) framework to exploit both interactions and words seamlessly. We learn such embeddings of users, items, and words jointly, and predict user preferences on items based on these learned representations. PNE estimates the probability that a user will like an item by two terms---behavior factors and semantic factors. On two real-world datasets, PNE shows better performance than four state-of-the-art baselines in terms of three metrics. We also show that PNE learns meaningful word embeddings by visualization.

Figures

Tables

Table 1. Table 1: Datasets and statistics.

Dataset	#user	#item	#rating	#word	#density	avg. words
Amazon	8,514	28,262	56,050	1,845,387	0.023%	65.3
Cheetah	15,890	84,802	477,685	612,839	0.035%	7.2

Table 2. Table 2: Categorization of recommender approaches.

Baselines	Shallow method	Deep method
CF	BPRMF	MLP
CF w/ Text	HFT, TBPR	LCMR, PNE (ours)

Table 3. Table 3: Results ( × 100 absent 100 \times 100 ) on Amazon dataset. Best baseline marked with asterisk and best result in boldfaced.

TopK	Metric	Method
		BPR	HFT	TBPR	MLP	LCMR	PNE
5	HR	8.10	10.77	15.17	21.00*	20.24	23.52
	NDCG	5.83	8.15	12.08	14.86*	14.51	16.46
	MRR	5.09	7.29	11.04	12.83*	12.63	14.13
10	HR	12.04	13.60	17.77	28.36*	28.36*	31.86
	NDCG	7.10	9.07	12.91	16.97*	16.78	19.15
	MRR	5.61	7.67	11.38	13.71*	13.56	15.24
20	HR	18.21	27.82	22.68	38.20	39.51*	42.21
	NDCG	8.64	12.52	14.14	18.99	19.18*	21.75
	MRR	6.02	8.54	11.71	14.26*	14.20	15.95

Table 4. Table 4: Results ( × 100 absent 100 \times 100 ) on Cheetah dataset. Best baseline marked with asterisk and best result in boldfaced.

TopK	Metric	Method
		BPR	HFT	TBPR	MLP	LCMR	PNE
5	HR	43.80	49.66	49.48	53.80	54.76*	56.48
	NDCG	39.71	36.17	42.98*	41.21	41.89	43.45
	MRR	36.06	31.75	38.26*	37.02	37.62	39.11
10	HR	49.41	55.80	54.66	61.76	63.11*	64.24
	NDCG	41.82	40.93	44.99*	43.81	44.60	45.98
	MRR	36.94	33.65	39.13*	38.10	38.74	40.16
20	HR	53.98	65.47	61.23	67.93	69.27*	69.52
	NDCG	43.16	43.79	46.82*	45.29	46.19	47.32
	MRR	37.30	34.45	39.58*	38.51	39.18	40.53

Equations

$${\hat{y}}_{ui}=f(u,i|\Theta ),$$

$${\mathbf{z}}_{ui}=\text{ReLU}(\mathbf{W}{\mathbf{x}}_{ui}+\mathbf{b}),$$

$${\mathbf{o}}_{ui}={\sum }_{j:w_j\in d_{ui}}\text{Softmax}(a_j^{u,i}){\mathbf{c}}_j,$$

$$a_j^{u,i}={\mathbf{x}}_{ui}^T{\mathbf{m}}_j^{u,i},$$

$${\tilde{\mathbf{z}}}_{ui}=[{\mathbf{z}}_{ui},{\mathbf{o}}_{ui}].$$

$${\hat{y}}_{ui}=\frac{1}{1+\exp (-{\mathbf{h}}^T{\tilde{\mathbf{z}}}_{ui})},$$

$$\mathcal{L}(\Theta )=-\sum _{(u,i)\in \mathcal{S}}{y}_{ui}\log {\hat{y}}_{ui}+(1-y_{ui})\log (1-{\hat{y}}_{ui}),$$

Full text

1 Introduction

Recommender systems are widely used in e-commerce platforms, such as to help consumers buy products at Amazon, watch videos on Youtube, and read articles on Google News. They are useful to alleviate the information overload and improve user satisfaction. Given history records of consumers such as the product transactions and movie watching, collaborative filtering (CF) is among the most effective approaches based on the simple intuition that if users rated items similarly in the past then they are likely to rate items similarly in the future Sarwar et al. (2001).

History records include both implicit (e.g., purchase and clicks) and explicit (e.g., likes/dislikes and ratings) feedback which can be represented as a user-item interaction matrix. Typically, observed user-item interactions are incomplete with a large portion remaining not recorded. The goal of recommendation is to predict user preferences on these missing interactions. This setting requires to complete the partial observed rating matrix. Matrix Factorization (MF) techniques which can learn latent factors for users and items are the main cornerstone for CF Mnih and Salakhutdinov (2008); Koren (2008); Koren et al. (2009). It is effective and flexible to integrate with additional data sources Hu et al. (2015). Recently, neural networks like Multilayer Perceptron (MLP) are used to learn an interaction function from data with the power of learning highly nonlinear relationships between users and items Dziugaite and Roy (2015); Cheng et al. (2016); He et al. (2017); Hu et al. (2018b). MF and neural CF exploit user-item behavior interactions only and hence they both suffer from the data sparsity and cold-start issues.

Items are usually associated with unstructured text, like news articles and product reviews. These additional sources are essential for recommendation beyond user-item interactions since they contain independent and diverse information. Hence, they provide an opportunity to alleviate the data sparsity issue Ganu et al. (2009); Hu et al. (2018a). For application domains like recommending research papers and news articles, the unstructured text associated with the item is its text content Wang and Blei (2011); Wang et al. (2015); Bansal et al. (2016). For some domains like recommending products, the unstructured text associated with the item is its user reviews which justify the rating behavior McAuley and Leskovec (2013); He and McAuley (2016). These methods adopt topic modelling techniques and neural networks to exploit the item content leading to performance improvement.

A typical way of exploiting text content is to firstly extract a feature vector for each document by averaging word embeddings in the document, and then to learn a text factor corresponding to this feature vector Hu and Dai (2017). These embeddings are pre-trained from a large corpus such as Wikipedia. This approach separates the extraction of text feature from the learning of user-item interaction. These two processes cannot benefit from each other and errors in the previous step maybe propagate to the successive steps. Another way is to learn a topic vector using topic modelling Wang and Blei (2011); McAuley and Leskovec (2013); Bao et al. (2014) by aligning behavior factors and topic factors with a link function such as softmax and offset.

Recently, neural networks are used to learn a representation from the text using autoencoders Wang et al. (2015); Zhang et al. (2016), recurrent networks Bansal et al. (2016), and convolutional networks Zheng et al. (2017); Catherine and Cohen (2017). These methods treat different words in the document as equal importance and do not match word semantics with the specific user. Instead, we achieve to learn a personalized word embedding with the guidance of user-item interactions. That is, the importance of words is learned to match user preferences. The attention mechanism can be used to learn these importance weights. Memory Networks (MemNet) have been used in recommendation to model item content Hu et al. (2018c); Huang et al. (2017), capture user neighborhood Ebesu et al. (2018), and learn latent relationships Tay et al. (2018). We follow this thread to adapt a MemNet to match word semantics with user preferences.

In this paper, we propose a novel neural framework to exploit relational interactions and text content seamlessly. The proposed Personalized Neural Embedding (PNE) model fuses semantic representations learnt from unstructured text with behavior representations learnt from user-item interactions jointly for effective estimation of user preferences on items. PNE estimates the preference probability by two kinds of factors. The behavior factor is to capture the personalized preference of a user to an item learned from behavior interactions. The semantic factor is to capture the high-level representation attentively extracted from the unstructured text by matching word semantics with user preferences.

To model the behavior factor, we adopt a neural CF approach, which learns the user-item nonlinear interaction relationships using a neural network (CFNet). To model the semantic factor, we adopt a memory network to match word semantics with the specific user via the attention mechanism inherent in the memory module (MemNet), determining which words are highly relevant to the user preferences. PNE integrates relational interactions with unstructured text by bridging neural CF and memory networks. PNE can also learn meaningful word embeddings.

2 Approach

We present PNE to jointly learn representations of users, items, and words. PNE seamlessly captures nonlinear user-item interaction relationships and matches word semantics with user preferences.

Denote the set of users by $\mathcal{U}$ and items by $\mathcal{I}$. We use a rating matrix $\bm{Y}\in\mathbb{R}^{|\mathcal{U}|\times|\mathcal{I}|}$ to describe user-item interactions where each entry $y_{ui}\in\{0,1\}$ is 1 (observed entries) if user $u$ has an interaction with item $i$ and 0 (unobserved entries) otherwise. Usually the interaction matrix is very sparse since a user $u\in\mathcal{U}$ only consumed a very small subset of all items. For the task of item recommendation, each user is only interested in identifying $topK$ items (typically $K$ is small e.g. tens or hundreds). The items are ranked by their predicted scores:

[TABLE]

where $f$ is the interaction function and $\Theta$ denotes model parameters.

2.1 Architecture

PNE consists of a CF network (CFNet) to learn a nonlinear interaction function and of a memory network (MemNet) to match word semantics with user preferences. The information flow in PNE goes from the input $(u,i)$ to the output $\hat{y}_{ui}$ through the following five modules.

1. Input: $(u,i)\rightarrow\vec{e}_{u},\vec{e}_{i}$ This module encodes user-item interaction indices. We adopt the one-hot encoding. It takes user $u$ and item $i$, and maps them into one-hot encodings $\vec{e}_{u}\in\{0,1\}^{|\mathcal{U}|}$ and $\vec{e}_{i}\in\{0,1\}^{|\mathcal{I}|}$ where only the element corresponding to that user/item index is 1 and all others are 0.

2. Embedding: $\vec{e}_{u},\vec{e}_{i}\rightarrow\bm{x}_{ui}$ This module firstly embeds one-hot encodings into continuous representations $\bm{x}_{u}=\bm{P}^{T}\vec{e}_{u}$ and $\bm{x}_{i}=\bm{Q}^{T}\vec{e}_{i}$ by embedding matrices $\bm{P}\in\mathbb{R}^{|\mathcal{U}|\times d}$ and $\bm{Q}\in\mathbb{R}^{|\mathcal{I}|\times d}$ respectively, where $d$ is the latent dimension. It then concatenates them as $\bm{x}_{ui}=[\bm{x}_{u},\bm{x}_{i}]$ to be the input of following CFNet and MemNet modules.

3. CFNet: $\bm{x}_{ui}\rightarrow\bm{z}_{ui}$ This module is a CF approach to exploit user-item interactions. It takes continuous representations from the embedding module and then transforms to a final behavior factor representation:

[TABLE]

where $\textrm{ReLU}(x)=\max(0,x)$ is an activation function, and $\bm{W}$ and $\bm{b}$ are connection weights and biases.

4. MemNet: $\bm{x}_{ui}\rightarrow\bm{o}_{ui}$ This module is to model the item content with the guidance of user-item interaction. The item content is modelled by memories. It takes representations from both the embedding module and the review text $d_{ui}$ associated with the corresponding user-item $(u,i)$ into a final semantic factor representation:

[TABLE]

where the external memory slot $\bm{c}_{j}$ is an embedding vector for word $w_{j}$ by mapping it with an external memory matrix $\bm{C}$. The attentive weight $a_{j}^{u,i}$ encodes the relevance of user $u$ to word $w_{j}$ by content-based addressing:

[TABLE]

where memory $\bm{m}_{j}^{u,i}$ is concatenated from internal memory slots $\{\bm{m}_{j}^{u},\bm{m}_{j}^{i}\}$ which are mapped from word $w_{j}$ by internal memory matrices $\bm{A}^{u}$ for user attention and $\bm{A}^{i}$ for item attention.

5. Output: $[\bm{z}_{ui},\bm{o}_{ui}]\rightarrow\hat{y}_{ui}$ This module predicts the recommendation score $\hat{y}_{ui}$ for a given user-item pair based on the representation of both behavior factor and semantic factor from CFNet and MemNet respectively:

[TABLE]

The output is the probability that the input pair is a positive interaction. This is achieved by a logistic layer:

[TABLE]

where $\bm{h}$ is model parameter.

2.2 Learning

We adopt the binary cross-entropy loss:

[TABLE]

where $\mathcal{S}=\bm{Y}^{+}\cup\bm{Y}^{-}$ is the union of observed interactions and randomly sampled negative examples. Model parameters $\Theta=\{\bm{P},\bm{Q},\bm{W},\bm{b},\bm{A},\bm{h}\}$ where we use a single word embedding matrix $\bm{A}$ by sharing all memory matrices $\bm{A}^{u},\bm{A}^{i}$, and $\bm{C}$ in order to reduce model complexity. The objective function can be optimized by stochastic gradient descent.

3 Experiment

In this section, we evaluate PNE on two datasets with five baselines in terms of three metrics.

3.1 Datasets

We evaluate on two real-world datasets. The public Amazon products  McAuley and Leskovec (2013) and a company Cheetah Mobile news Hu et al. (2018c); Liu et al. (2018) (see Table 1). We preprocess the data following the strategy in  Wang and Blei (2011). The size of word vocabulary is 8,000.

3.2 Metrics and Baselines

We adopt leave-one-out evaluation Hu et al. (2018b) and use three ranking metrics: hit ratio (HR), normalized discounted cumulative gain (NDCG), and mean reciprocal rank (MRR).

We compare with five baselines (see Table 2).

• •

BPR Rendle et al. (2009) is a latent factor model based on matrix factorization.

• •

HFT McAuley and Leskovec (2013) adopts topic distributions to learn latent factors from text reviews.

• •

TBPR Hu and Dai (2017) extends BPR by integrating text content via word embedding features. The word embeddings used in the TBPR are pre-trained by GloVe Pennington et al. (2014).

• •

MLP He et al. (2017) is a neural CF approach. Note that, CFNet in PNE is an MLP with only one hidden layer.

• •

LCMR Hu et al. (2018c) is a deep model for CF with unstructured text.

Our method is implemented by TensorFlow Abadi et al. (2016). Parameters are randomly initialized from Gaussian with optimizer Adam Kingma and Ba (2015). Learning rate is 0.001, batch size is 128, the ratio of negative sampling is 1.

3.3 Results

Results on two datasets are shown in Table 3 and Table 4. We have some observations. First, PNE outperforms the neural CF method MLP on two datasets in terms of three ranking metrics. On Amazon dataset, PNE obtains a large improvement in performance gain with relative 12.3% HR@10, 7.7% NDCG@10, and 6.2% MRR@5. On Cheetah Mobile dataset, PNE obtains a large improvement in performance gain with relative 5.0% HR@5, and 4.2% NDCG@5, and 3.9% MRR@5. Since the CFNet component of PNE is a neural CF method (with only one hidden layer), results show the benefit of exploiting unstructured text to alleviate the data sparsity issue faced by CF methods (BPR and MLP).

Second, PNE outperforms the traditional hybrid methods HFT and TBPR on two datasets in terms of three ranking metrics. On Amazon dataset, PNE obtains a significantly large improvement in performance gain with relative 55.0% HR@5, 28.9% NDCG@5, and 20.4% MRR@5. On Cheetah Mobile dataset, PNE still obtains reasonably large improvements with relative 17.5% HR@10, 1.8% NDCG@10, and 1.9% MRR@10. Compared with traditional hybrid methods which integrate the text content using topic modelling or word embeddings, results show the benefit of integrating text information through memory networks (and exploiting the interaction data through neural CF).

Last, PNE outperforms neural hybrid method LCMR by a large margin on Amazon dataset with relative improvements of 16.2% HR@5, 9.6% NDCG, and 7.4% MRR@5. PNE obtains reasonable improvements on Cheetah Mobile dataset with relative improvements of 3.1% HR@5, 2.8% NDCG, and 2.7% MRR. The design of CFNet of PNE is more reasonable than that of centralized memory module of LCMR which is equivalent to use a softmax activation between two hidden layers. The results show the effectiveness of fusing strategy in PNE to exploit unstructured text via MemNet and the interaction data via CFNet.

3.4 Analysis

We first evaluate the effects of the dimensionality of the embedding space. The $x$-axis in Figure 1 is the dimension of user/item and hence the dimensionality of input to CFNet is double since we adopt concatenation. It clearly indicates that the embedding should not be too small due to the possibility of information loss and the limits of expressiveness. The dimension 75 (and hence $d=150$) is a good tradeoff between recommendation performance and computation burden.

We next show optimization curves of performance and loss (averaged over all examples) against iterations on Cheetah Mobile dataset in Figure 2. The model learns quickly in the first 20 iterations and improves slowly until 50, while training losses continue to go down and valid losses stabilize. The average time per epoch of PNE takes 68.1s and as a reference it is 34.5s for MLP using one NVIDIA TITAN Xp GPU.

3.5 Visualization

We visualize learned word embeddings $\bm{A}$. We show that we can learn meaningful semantics for word embeddings such that words are to cluster when they have relevant semantics. We give an example to show the neighbors of the word “drug” in the 3D space by projecting the high-dimensional word vectors using TensorFlow 111https://projector.tensorflow.org/ as shown in Figure 3. The top nearest neighbors of drug are: shot, shoots, gang, murder, killing, rape, stabbed, truck, school, police, teenage. We can see they are highly semantic relevant. We may also infer that school teenagers have close relationships to the drug issue from the Cheetah News corpus. This should raise a concern for the whole society and it shows the society impact of natural language processing Hovy and Spruit (2016). Try it here: https://www.dropbox.com/sh/ef74fpagf6sd137/AACXF6EnEY6QBdmJcyIW4RE0a?dl=0.

4 Conclusion

We showed that relational interactions can be effectively integrated with unstructured text under a neural embedding model. Our method attentively focuses relevant words to match user preferences with user and item attentions (semantic factor) and captures nonlinear relationships between users and items (behavior factor). Experiments show better performance than five baselines on two real-world datasets in terms of three ranking metrics. We learn meaningful word embeddings and rethink the society impact of language processing technology.

Acknowledgment The work is supported by HK CERG projects 16211214/16209715/16244616, NSFC 61673202, and HKPFS PF15-16701.