Herding Effect based Attention for Personalized Time-Sync Video Recommendation

TL;DR

This paper introduces ITF-HEA, a novel model leveraging herding effect attention on time-sync comments and video images to improve personalized video recommendations, outperforming existing methods.

Contribution

The paper proposes a new Image-Text Fusion model with Herding Effect Attention to incorporate real-time, context-dependent TSC data into video recommendation systems.

Findings

ITF-HEA achieves 3.78% higher F1-score than state-of-the-art methods.

The model effectively captures herding effects in TSC data.

Incorporating time and semantic similarities improves recommendation accuracy.

Abstract

Time-sync comment (TSC) is a new form of user-interaction review associated with real-time video contents, which contains a user's preferences for videos and therefore well suited as the data source for video recommendations. However, existing review-based recommendation methods ignore the context-dependent (generated by user-interaction), real-time, and time-sensitive properties of TSC data. To bridge the above gaps, in this paper, we use video images and users' TSCs to design an Image-Text Fusion model with a novel Herding Effect Attention mechanism (called ITF-HEA), which can predict users' favorite videos with model-based collaborative filtering. Specifically, in the HEA mechanism, we weight the context information based on the semantic similarities and time intervals between each TSC and its context, thereby considering influences of the herding effect in the model. Experiments show that ITF-HEA is on average 3.78\% higher than the state-of-the-art method upon F1-score in baselines.

Figures

Tables

Table 1. Table 1 : Precision and F1-score of each method

	Top5		Top10		Top20
	Prec	F1	Prec	F1	Prec	F1
HFK	0.856	0.3463	0.812	0.5310	0.732	0.7371
JMARS	0.878	0.3552	0.810	0.5560	0.732	0.7367
VBPR	0.892	0.3608	0.83	0.5760	0.779	0.7840
KFRCI	0.954	0.3859	0.897	0.6036	0.818	0.8233
ITF-HEA	0.976	0.3948	0.932	0.6272	0.860	0.8656

Table 2. Table 2 : Precision and F1-score of the models proposed in Section 3

	Top5		Top10		Top20
	Prec	F1	Prec	F1	Prec	F1
TM	0.932	0.2363	0.887	0.5969	0.790	0.7950
T-HEA	0.956	0.3867	0.914	0.6151	0.817	0.8223
ITF	0.952	0.3851	0.892	0.6003	0.805	0.8107
ITF-HEA	0.976	0.3948	0.932	0.6272	0.860	0.8656

Equations

$${\overrightarrow{\mathbf{h}}}_{\mathbf{t}}=LSTM(w_i^t,{\overrightarrow{\mathbf{h}}}_{\mathbf{t}-1})$$

$${\overleftarrow{\mathbf{h}}}_{\mathbf{t}}=LSTM(w_i^t,{\overleftarrow{\mathbf{h}}}_{\mathbf{t}+1})$$

$$\mathbf{se}{\mathbf{q}}_{\mathbf{i}}=\frac{1}{L_i}\sum _{t=1}^{L_i}(\overrightarrow{{\mathbf{h}}_t}\oplus \overleftarrow{{\mathbf{h}}_t})$$

$${\mathbf{p}}_{\mathbf{i}}={\mathbf{G}}_{{\mathbf{u}}_{\mathbf{i}}}\otimes \mathbf{se}{\mathbf{q}}_{\mathbf{i}}$$

$${\mathbf{q}}_{\mathbf{i}}={\mathbf{G}}_{{\mathbf{v}}_{\mathbf{i}}}\otimes \mathbf{se}{\mathbf{q}}_{\mathbf{i}}$$

$$(a_1,...,a_d)\otimes (b_1,...,b_d)=(a_1{b}_1,...,a_d{b}_d)$$

$${\hat{y}}_i=sigmoid({\mathbf{p}}_{\mathbf{i}}\odot {\mathbf{q}}_{\mathbf{i}})$$

$$\begin{array}{rl}L=& \sum _{i=1}^N(y_i\cdot ln{\hat{y}}_i+(1-y_i)\cdot ln(1-\widehat{y_i}))\end{array}$$

$${\hat{y}}_{u,v}=\mathbf{G}{\mathbf{U}}_{\mathbf{u}}\odot \mathbf{G}{\mathbf{V}}_{\mathbf{v}}$$

$$P{o}_{u,v}=\frac{{\sum }_{i\in Lis{t}_{u,v}}po{l}_i}{|Lis{t}_{u,v}|}$$

$$y_{u,v}=\{\begin{array}{ll}0& P{o}_{u,v}<0.5\\ 1& P{o}_{u,v}\ge 0.5\end{array}$$

$$\mathbf{vs}{\mathbf{l}}_{\mathbf{i}}^{\prime }=Dense(\mathbf{vs}{\mathbf{l}}_{\mathbf{i}})$$

$$\mathbf{co}{\mathbf{m}}_{\mathbf{i}}=\mathbf{se}{\mathbf{q}}_{\mathbf{i}}\oplus \mathbf{vs}{\mathbf{l}}_{\mathbf{i}}^{\prime }$$

$$\mathbf{co}{\mathbf{m}}_{\mathbf{i}}^{\prime }=Dense(\mathbf{co}{\mathbf{m}}_{\mathbf{i}})$$

$${\mathbf{h}}_{\mathbf{t}}=LSTM({\mathbf{h}}_{\mathbf{t}-1},\mathbf{se}{\mathbf{q}}_{\mathbf{i},\mathbf{t}})$$

$$sim(k,j)=\frac{\mathbf{se}{\mathbf{q}}_{\mathbf{i},\mathbf{k}}\odot \mathbf{se}{\mathbf{q}}_{\mathbf{i},\mathbf{j}}}{|\mathbf{se}{\mathbf{q}}_{\mathbf{i},\mathbf{k}}||\mathbf{se}{\mathbf{q}}_{\mathbf{i},\mathbf{j}}|}$$

$$delay(j,k)=\{\begin{array}{ll}{\mathrm{e}}^{-\beta (t_{i,j}-t_{i,k})}& j>k\\ 0& j<=k\end{array}$$

$${\mathbf{SI}{\mathbf{M}}_{\mathbf{j}}}^{\prime }=(\frac{{\mathrm{e}}^{sim(j,1)}}{{\sum }_{k=1}^M{\mathrm{e}}^{si{m}_{j,k}}},\frac{{\mathrm{e}}^{sim(j,2)}}{{\sum }_{k=1}^M{\mathrm{e}}^{si{m}_{j,k}}},...,\frac{{\mathrm{e}}^{sim(j,M)}}{{\sum }_{k=1}^M{\mathrm{e}}^{si{m}_{j,k}}})$$

$${\mathbf{A}}_{\mathbf{j}}={\mathbf{SI}{\mathbf{M}}_{\mathbf{j}}}^{\prime }\otimes \mathbf{T}{\mathbf{D}}_{\mathbf{j}}$$

$${\mathbf{C}}_{\mathbf{j}}={\overline{\mathbf{A}}}_{\mathbf{j}}\odot \mathbf{H}$$

$${\tilde{\mathbf{h}}}_1=LSTM({\mathbf{C}}_1)$$

$${\tilde{\mathbf{h}}}_{\mathbf{t}}=LSTM({\mathbf{C}}_{\mathbf{t}},{\tilde{\mathbf{h}}}_{\mathbf{t}-1})$$

Full text

Index Terms—  Recommendation System, Collaborative Filtering, Herding Effect, Data Mining

1 Introduction

Recently, watching online videos of news and amusement has become mainstream entertainment during people’s leisure time. Therefore, efficient and accurate personalized video recommendation methods bring significant convenience to their life. Most of the video recommendation methods focus on users’ behaviors such as their browsing history [1, 2] and reviews [3, 4]. In real scenarios, however, most people are unwilling to do high-quality reviews after watching videos, which causes the scarcity of valuable video reviews. Furthermore, some multi-feature based methods [5, 6] combine image information with review information to generate users’ interests from a more comprehensive perspective. However, their methods have only achieved limited improvement because the images and reviews usually contain unequal information [7]. That is, the text information and image information generally describe the different amount of contents. An image in a video only describes one moment of the video content, while a review usually describes the overall contents of the video. The information gap causes the fusion of reviews and images to lose great information.

Meanwhile, a new form of user-interactive review – time-sync comment (TSC) (first introduced by Wu et al. [8], see Fig. 1) has become increasingly popular in China and Japan, especially among young people. Nowadays, many popular Chinese video websites such as Youku (http://www.youku.com), Bilibili (http://bilibili.tv) and the Japanese video website NICONICO (http://www.nicovideo.jp) support the TSC. TSCs convey information involving the content of the current video frame, feelings of users or replies to other TSCs, which can accurately express the users’ preferences for the video. Moreover, each TSC has a corresponding timestamp to record the posted time. Compared with traditional video reviews, TSCs are much easier to obtain their corresponding images by timestamps. The users’ real-time feedbacks and the vast amount make TSCs valuable and accessible sources for personalized video recommendations.

In this paper, we focus on mining the users’ preferences and videos’ features from TSCs and corresponding images to recommend videos towards users through model-based Collaborative Filtering (CF). TSCs have several features distinguished from the traditional video reviews: (1)Context-dependent. TSCs are usually context-dependent, i.e., the latter comments often depend on the former ones. This phenomenon is known as the herding effect in social science [9, 10]. An example of the herding effect is shown in Fig. 1. User A said ”I love the male commander!” to express his love to the role male commander when he appears in the video. After a few seconds, user B and user C followed up by saying ”I like the male commander too…” and ”I am so sad when he died.” In this case, user B and user C may not make their comments if user A has not. That is, the emergence of a TSC is usually not independent, but a probability event influenced by other preorder comments. (2)Real-time. Each TSC has a timestamp synchronous to the playback time of the video. The coverage of a TSC is usually only a short time before its timestamp. Therefore, the content of each TSC is closely related to the video content corresponding to its timestamp, which makes it easy to sample corresponding image information by timestamp. (3)Time-sensitive. According to our observation, TSCs with a large interval of timestamps are unlikely to discuss the same topic, even if their semantics are similar. Users are more likely to follow those newer TSCs than older ones. As a result, the herding effect mentioned before will not last long. These features make TSC a particular review. However, most of the current TSC-based recommendations [3, 11] assume that TSCs are independent of each other and ignore the time information. Such assumption ignores the above features of TSCs which causes the loss of crucial semantic information and affects the accuracy of results. Therefore, how to take TSCs’ features of context-dependent (herding effect), real-time and time-sensitive into account to extract the textual information and fuse it with visual information accurately and effectively are the central challenges.

Based on the above motivations and challenges, we propose an Image-Text Fusion model with a novel Herding Effect Attention mechanism (called ITF-HEA). In ITF-HEA, We generate users’ preferences and summarize video contents through model-based CF. To analyze the influence of text information, image information and contextual information separately, we split ITF-HEA into two models: Text-based Model(TM) and Image-Text Fusion model (ITF), and one attention mechanism: Herding Effect Attention (HEA) mechanism. Specifically, in TM, we sample and embed the TSCs to obtain the sentence vectors (TSC features) by bidirectional Long Short-Term Memory (LSTM) at first, and then combine TSC features with the hidden (embedding) features of the users and videos respectively to predict the likeness of the user to the video. In ITF, we sample the corresponding video frame (image) features and incorporate them with TSC features to replace single TSC features in TM. Finally, we design the HEA mechanism which is based on contextual semantic similarity and time interval of TSCs to incorporate contextual information into TSC features to replace the features in TM and ITF.

The main contributions of this paper are as follows:

• 1

We propose a novel HEA mechanism, which takes TSCs’ features of contextual relevance, real time and timeliness into account, to extract the textual features of the TSCs more accurately and effectively.

• 2

We design an Image-Text Fusion model using model-based CF, combine it with HEA mechanism and get the ITF-HEA, which can predict the likeness of the user to the video more accurately and sufficiently.

• 3

We evaluate the ITF-HEA with real-world datasets on mainstream video-sharing websites and compare it with state-of-the-art video recommendation methods. The results show that our model outperforms baselines in both precision and F1-score on average 3.3% and 3.78% respectively.

2 Related work

In this section, we discuss the related work in three aspects.

Time-sync video comments are first introduced by Wu et al. [8]. Then, Yang et al. [12] sum up the features of TSCs, which inspire our work. LV et al. [13] propose a video understanding framework to assign temporal labels to highlighted video shots. They are the first to analyze the TSCs using the neural network. Recently, Liao et al. [14] present a larger-scale TSC dataset with four-level structures and rich self-labeled attributes, which brings convenience for future research on TSCs. The above methods show that TSC is a kind of data with great potential and development value.

Video recommendation has attracted great attention from both the industry and academia. Most of the current state-of-the-art methods are based on CF. Mcauley et al. [15] combine latent rating dimensions with latent review topics, which is a review-based method. Diao et al. [16] propose a probabilistic model based on CF and topic modeling, which is an LDA [17] based method and allows capturing the interest distribution of users and the content distribution of movies. He et al. [18] propose a scalable factorization model to incorporate visual signals into predictors of people’s opinions, which is the state-of-the-art visual-based model. However, the above recommendation methods are not well-designed for TSCs as they ignore the interactive, real-time, and timeliness properties of TSC data.

Attention Mechanism has been shown effective in natural language processing [19, 20, 21]. Recently, attention models have been used increasingly in recommendation systems to assign weights to user-item pairs. Chen et al. [22] introduce a novel attention mechanism in CF to address the challenging item and component-level implicit feedback in the multimedia recommendation, which can be seamlessly incorporated into classic CF models with implicit feedback. Seo et al. [23] propose to model user preferences and item properties using convolutional neural networks (CNN) with dual local and global attention. Our herding effect attention mechanism adopts the soft attention [24], which learns the attentive weights based on the importance to the final task.

3 Model-based Collaborative Filtering

In this section, we describe two CF models and an attention mechanism. First, the problem formulation is provided in Section 3.1. Then, we propose a Text-based Model by using textual features of TSCs in Section 3.2. Next, we design an Image-Text Fusion Model to jointly model video images as well as TSCs in Section 3.3. Finally, to take full consideration of the features of TSCs, we implement Herding Effect Attention mechanism and give the complete neural network structure of the Image-Text Fusion Model with Herding Effect Attention in Section 3.4.

3.1 Problem Formulation

Suppose there are $N$ TSCs, $\boldsymbol{TSC}$ $=$ $\{tsc_{1},tsc_{2},...,tsc_{N}\}$. For $tsc_{i}$, we define the corresponding visual feature as $\boldsymbol{vsl_{i}}$ (see Section 3.3 for details) and sentiment polarity $pol_{i}$ which is determined by the Stanford sentiment analysis toolkit (http://nlp.stanford.edu/sentiment). Besides, we define $u_{i}$ to represent the user ID and $v_{i}$ to express the video ID of $tsc_{i}$.

As mentioned in section 1, TSCs are easily affected by previous comments. Therefore, for $tsc_{i}$, we continuously sample M preorder TSCs $\boldsymbol{Context_{i}}=\{pre_{i,1},pre_{i,2},...,pre_{i,M}\}$ as context information ($pre_{i,M}$ is $tsc_{i}$ itself). For each $pre_{i,j}\in\boldsymbol{Context_{i}}$, we define the time-stamp $t_{i,j}$ to represent its posted video time. The word list of $tsc_{i}$ and its context information $pre_{i,j}$ are defined as $\boldsymbol{w_{i}}=\{w_{i}^{1},w_{i}^{2},...,w_{i}^{L_{i}}\}$ and $\boldsymbol{w_{i,j}}=\{w_{i,j}^{1},w_{i,j}^{2},...,w_{i,j}^{L_{i,j}}\}$ where $L_{i}$ and $L_{i,j}$ are the length of $tsc_{i}$ and $pre_{i,j}$.

Given total TSCs $\boldsymbol{WT}$ $=$ $\{\boldsymbol{w_{i}}|1\leq i\leq N\}$ and their corresponding sentiments $\boldsymbol{POL}$ $=$ $\{pol_{i}|1\leq i\leq n\}$, user ID list $\boldsymbol{U}$ $=$ $\{u_{1},u_{2},...,u_{N}\}$, video ID list $\boldsymbol{V}$ $=$ $\{v_{1},v_{2},...,v_{N}\}$, visual feature list $\boldsymbol{VSL}$ $=$ $\{vsl_{1},vsl_{2},...,vsl_{N}\}$, and the context information $\boldsymbol{WC}$ $=$ $\{\{\boldsymbol{w_{1,1},...,w_{1,M}}\},...,\{\boldsymbol{w_{N,1},...,w_{N,M}}\}\}$ with the corresponding time-stamp $\boldsymbol{T}$ $=$ $\{\{t_{1,1},...,t_{1,M}\},...,\{t_{N,1},...,t_{N,M}\}\}$, our task is to predict the likeness of user $u\in\boldsymbol{UID}$ on video $v\in\boldsymbol{VID}$, where $\boldsymbol{UID}$ and $\boldsymbol{VID}$ are the sets that contain the unique IDs in $\boldsymbol{U}$ and $\boldsymbol{V}$. The top $X$ videos among the final results are recommended to the corresponding user.

3.2 Text-based Model

Intuitively, the preference of users is extracted from their published TSCs in corresponding videos, while the textual features of the videos are summed up from the TSCs published in the video. Based on above, in this section, we first extract the features of TSCs by bidirectional LSTM. Then, features of TSCs are merged with the latent factors of users and videos. Finally the likenesses of the users to videos are predicted by CF. The general framework of Text-based Model (TM) is shown in Fig. 2.

More concretely, to capture the word sequential information from TSCs, we use the Bidirectional Long Short-Term Memory (Bi-LSTM) network [25] to convert word features into TSC features.

For each $tsc_{i}$, we have

[TABLE]

and

[TABLE]

where $\boldsymbol{seq_{i}}\in\mathbb{R}^{d}$ and $\oplus$ denotes vector concatenation. After LSTM layer, we get the sequence feature $\boldsymbol{seq_{i}}$ as the output.

Then, we define $\boldsymbol{GU_{u_{i}}}$ as the latent factor of user $u_{i}$, which is the feature based on user’s historical preference. Likewise, the feature of video $v_{i}$ is defined as $\boldsymbol{GV_{v_{i}}}$. Afterward, we design $\otimes$ function to merge $\boldsymbol{GU_{u_{i}}}$ and $\boldsymbol{GV_{v_{i}}}$ with $\boldsymbol{seq_{i}}$ respectively, and obtain

[TABLE]

and

[TABLE]

where $\otimes$:$R^{d}\times R^{d}->R^{d}$ is an element-wise product function to merge two $d$ dimensional vectors into one. Specifically,

[TABLE]

In our framework, we take the prediction of a user’s favor to a video (or video clip) as a binary classification problem, where 1 means a user likes the video, and 0 otherwise. Therefore, we define the likeness of user $u_{i}$ to video $v_{i}$ though $tsc_{i}$ in the training data as

[TABLE]

where $sigmoid(x)=\frac{1}{1+e^{-x}}$ and “$\odot$” denotes inner product.

Generally, users comment on their favorite videos with positive sentiment. Therefore, we determine the polarity of each TSC by the Stanford sentiment analysis toolkit [26]. For simplicity, we set the polarity of each TSC as 1 if the result is positive or neutral, and 0 otherwise. We define $y_{i}=pol_{i}$ as the ground truth of the likeness of user $u_{i}$ for video $v_{i}$ through $tsc_{i}$, where $pol_{i}$ is the polarity of $tsc_{i}$.

At last, we use the binary cross-entropy as our loss function to model user preference. The final objective function is maximized as:

[TABLE]

In the training phase, the parameters can be learned via Adam [27].

After trainning, we use

[TABLE]

to express the predicted likeness of user $u$ on video $v$, and

[TABLE]

to express the real likeness of user $u$ for video $v$, where $List_{u,v}$ expresses the total TSCs that user $u$ has commented on video $v$. Then, the ground truth of testing data is defined as

[TABLE]

3.3 Image-Text Fusion Model

In the time-sync video, each TSC has a timestamp that records the corresponding video time when the TSC is published. So that, we can easily obtain the corresponding image information for better feature extraction. In this section, we focus on merging TSC text features with corresponding visual features to obtain more comprehensive features. The general framework of Image-Text Fusion model (ITF) is shown in Fig. 3.

For $tsc_{i}$, we use $vsl_{i}$ to indicate the visual feature when $tsc_{i}$ is posted. The visual features are with the output of 4096-way obtained from a public TSC data set extracted by Chen et al. [3], which are trained by the Caffe reference model with 5 convolutional layers followed by 3 fully-connected layers that have been pre-trained on 1.2 million ImageNet (ILSVRC2010) images.

Since the dimension of $\boldsymbol{vsl_{i}}$ is 4096, we reduce its dimension to $d$ and get $\boldsymbol{vsl_{i}^{\prime}}$ by

[TABLE]

where $Dense$ is the fully-connected layer with the activation function $elu$ [28].

To combine the image features and textual features, we first concatenate the sequence feature $seq_{i}$ with the visual feature $vsl_{i}^{\prime}$, and obtain the $2\times{d}$-dimensional vector $com_{i}$:

[TABLE]

Then, we reduce the demension of $com_{i}$ to $d$ and get the $com_{i}^{\prime}$ by

[TABLE]

Finally, we use $com_{i}^{\prime}$ instead of $seq_{i}$ to merge with $\boldsymbol{GU_{u_{i}}}$ and $\boldsymbol{GV_{v_{i}}}$ by $Eq.(\ref{eq:merge1})$ and $Eq.(\ref{eq:merge2})$ and predict the likeness by $Eq.(\ref{eq:pred})$.

3.4 Herding Effect Attention

Existing review-based recommendation methods usually handle each comment separately without considering the context associations between the comments. However, TSCs are highly semantic relevant and time-related, which is so-called the herding effect. That is, TSCs may be affected by other preorder TSCs on the similar topic. Also, TSCs with similar semantics and the short interval of the time-stamp are more likely to influence each other. Based on above, we design an HEA mechanism, which calculates the influence weights of TSC contexts by their semantic similarities and timestamp intervals in an LSTM-based encoder-decoder framework. The framework of HEA is shown in Fig. 4.

Formally, for $tsc_{i}$, we sample $M$ continuous preorder TSCs $\boldsymbol{Context_{i}}=\{pre_{i,1},pre_{i,2},...,pre_{i,M}\}$ as the context information and get context features $\boldsymbol{SEQ_{i}}=[\boldsymbol{seq_{i,1},...,seq_{i,M}}]$ by $Eq.(\ref{first})$ - $(\ref{seq})$, where $t_{i,1}<t_{i,2}<....<t_{i,M}$.

We formalize the HEA into an encoder-decoder framework. Given context features $\boldsymbol{SEQ_{i}}$ as the input to the LSTM, the output are obtained as:

[TABLE]

We use $\boldsymbol{H}=(\boldsymbol{h_{1},h_{2},...,h_{M}})$ to express the hidden status vectors of the encoder output.

To calculate the influence weights of contexts of TSCs, for $pre_{i,j}$, we define the semantic similarity vector and time delay vector as $\boldsymbol{SIM_{j}}=(sim(j,1),sim(j,2),...,sim(j,M))$ and $\boldsymbol{TD_{j}}=(delay(j,1),delay(j,2),...,delay(j,M))$ , where

[TABLE]

represents the semantic similarity between $pre_{i,j}$ and $pre_{i,k}$, and

[TABLE]

represents the influence of $pre_{i,k}$ on $pre_{i,j}$ decreases with the increasing time interval ($\beta$ is a hyper-parameter that will be discussed in Section 4).

Since the semantic similarity may have negative numbers, we first normalize $\boldsymbol{SIM_{j}}$ by softmax as:

[TABLE]

Next, we calculate the attention score vector of $pre_{i,j}$ as:

[TABLE]

The final attention score distribution $\boldsymbol{\overline{A}_{j}}$ is obtained by normalizing the attention score vector $\boldsymbol{A_{j}}$ by softmax function.

We compute the input of decoder as:

[TABLE]

and get the output as:

[TABLE]

[TABLE]

Finally, we use $\boldsymbol{\widetilde{h}_{M}}$ instead of $\boldsymbol{seq_{i}}$ that we used in Section 3.2 and 3.3 as the textual feature.

We integrate the context-dependent, real-time and time-sensitive properties of the TSCs into the model by the HEA mechanism, which can be applied in both TM and ITF to improve the TSC feature extraction. The complete network structure of the ITF-HEA is shown in Fig. 5.

4 Experiments

In this section, we demonstrate the effectiveness of our proposed method by comparing with 4 well-known methods of video recommendation. We provide necessary parameters of our model at first and then analyze the performance of our model on time-sync video recommendation. Finally, we analyze the effect of the hyper-parameters on the experimental results.

4.1 Experimental Setup and Dataset

The data used in this paper are crawled from a Chinese time-sync video site Bilibili by Chen et al. [3], which are obtained from the movie category till December 10th, 2015. In this paper, we select 100 users who have posted the most TSCs and commented on more than 40 videos. These users have commented on a total of 871 videos, and we select all the comments in those videos as a sub-dataset. In the sub-dataset, 423,384 users have published 1,319,475 TSCs in total.

For each of the 100 users, we select half of the videos where they have commented as the training set, and the other half as the test set. We make sure that at least 20 videos per user can be recommended (To ensure the effectiveness of top20). In the test set, we get 2,995 $(user,video)$ pairs, where 1,972 pairs are positive, and 1,023 pairs are negative in sentiment polarity.

In the training set, we obtain 2,811 $(user,video)$ pairs with 11,775 TSCs (a user may make more than one TSC in a video), where 8,124 TSCs are positive and 3,651 TSCs are negative.

In our model, hyper-parameter $\beta$ and number of contextual TSCs $M$ need to be decided. We select 35% data of the test set (actually 1,075 $(user,video)$ pairs) as the validation set to tune $\beta$. The initial learning rate of Adam [27] is 0.001 and the vector dimension $d$ is set as 128. We get the best results when $\beta=0.2$, and $M=10$, which are discussed in Section 4.2.

4.2 Results

In this section, we use the test set described in Section 4.1 to compare our complete model with existing methods.

To evaluate the performance of the proposed models, we compare our model with the following methods as baselines:

• •

HFT: A state-of-the-art method regarding making rating prediction with textual reviews [15]. In the experiments, we set the ratings of positively commented videos as 1, and 0 otherwise.

• •

JMARS: A Latent Dirichlet Allocation (LDA) based method to make rating prediction with textual reviews [16].

• •

VBPR: A visual-based recommendation method [18].

• •

KFRCI: A novel Key Frame Recommender by modeling user TSCs and keyframe Images simultaneously [3]. In the experiments, the likeness score of the video is the average score of all the frames the users have commented on.

• •

ITF-HEA: The Image-Text Fusion Model proposed in Section 3.3 with Herding Effect Attention mechanism proposed in Section 3.4.

For each method in the baselines, we select a set of the best experimental parameters according to the range of the parameters given in their experiments and calculate the likenesses/ratings between users and videos.

Our experiments are conducted by predicting Top 5, 10, and 20 favorite videos respectively. The Top $X$ is the top prediction of user’s likeness to the videos in test set calculated by Eq. (8). We recommend all X videos to each user and consider these are the user’s favorite videos. We adopt F1-score and precision to evaluate the performance of the baselines and our models. All the models are repeated for 10 times, and we report the average values as the final results for clear illustration.

The results of F1-score and precision are shown in Table 1. From Table 1, we can see: ITF-HEA achieves the best performance on F1 and precision (F1 is proportional to precision in the Top 5, 10 and 20). It has enhanced the performance by about $2.30\%$, $3.91\%$ and $5.14\%$ (on average 3.78%) upon F1-score and 2.20%, 3.50% and 4.20% (on average 3.30%) upon Precision on Top5, 10 and 20 respectively compared with KFRCI, which performs best among baselines. In other methods of baselines, the vision-based method VBPR has better performance than the others; the text-based method HFT and JMARS have similar performance, while the PMF method has the worst.

Next, we compare the models proposed in Section 3:

• •

TM: The Text-based Model proposed in Section 3.2.

• •

T-HEA: The Text-based Model proposed in Section 3.2 with Herding Effect Attention mechanism proposed in Section 3.4.

• •

ITF: The Image-Text Fusion Model proposed in Section 3.3.

• •

ITF-HEA: The Image-Text Fusion Model proposed in Section 3.3 with Herding Effect Attention mechanism proposed in Section 3.4.

to analyze the effects of text features, image features and the attention mechanism in our model. The results of F1-score and precision are shown in Table 2.

The results show that although T-HEA only uses the textual information, the experimental results are still better than ITF. T-HEA even has better performance than the state-of-the-art method KFRCI, which shows that our HEA mechanism can effectively offset the effects of the herding effect and improve the performance of the model. The results also show that the context and timestamp of the TSCs are vital information and need to be considered.

Finally, we discuss the influence of hyper-parameter $\beta$ and the number of contextual TSCs $M$ on the experimental results. We fix $M=10$, changing the value of $\beta$ from 0 to 0.5 (the step size is 0.1), and calculate the F1-score of Top 5,10 and 20 users’ favorite videos in the validation set at first. The best results are obtained when $\beta=0.2$. We also calculate the F1-score for the different hyper-parameters in the test set, and the results are shown in Fig. 7. The hyper-parameter $\beta$ gains the best performance when $\beta=0.2$ in any case, which is the same with the validation set. When $\beta$ is bigger, the result of the experiment is worse because it weakens the weight of other TSCs in the attention layer. When $\beta=0$, it has the worst performance, because the time information is not considered.

For the number of context length $M$, we fix $\beta=0.2$, and set $M$ as 5, 10, 15 and 20, respectively. The results are shown in Fig. 7. IFT-HEA gets the best performance when $M=10$ and the worst when $M=20$. This result confirms that the herding effect of TSCs is time-sensitive and will not last long, which meets our observation in Section 1.

5 Conclusion and Future Work

In this paper, we proposed a novel personalized online video recommendation with the dataset of both TSCs and its corresponding images through model-based CF method. To extract the textual features of the TSCs more accurately and effectively, we designed the HEA mechanism to add influence weight to each TSC based on their semantic similarity and time interval. In this way, we integrated the context-dependent, real-time and time-sensitive properties of TSCs in the neural network framework and predicted the users’ preferences for online videos accurately and effectively. Extensive experiments on real-world dataset proved that our model could recommend videos to users more precisely than the state-of-the-art method with the HEA mechanism.

This is the first step towards our goal in personalized video recommendation, and there is much space for further improvements. For example, to design a more accurate fusion model to capture comprehensive user preference is a challenging problem. Besides, how to measure the weight of each video to the user’s preference is also a challenging problem.

6 Acknowledgements

This work is supported by National China 973 Project No. 2015CB352401; Chinese National Research Fund (NSFC) Key Project No. 61532013 and No. 61872239. 0007/2018/A1, DCT-MoST Joint-project No. 025/2015/AMJ,FDCT,SAR Macau, China, and University of Macau Grant Nos: MYRG2018-00237-RTO, CPG2019-00004-FST and SRG2018-00111-FST.