Geometric Matrix Completion with Deep Conditional Random Fields

TL;DR

This paper introduces a deep conditional random field approach for geometric matrix completion that learns similarities and predicts missing entries effectively, especially in scenarios with very limited observations and no side information.

Contribution

It presents a novel end-to-end deep model that jointly learns similarities, computes CRF potentials, and performs MAP inference for matrix completion without requiring fixed graphs or side information.

Findings

Outperforms state-of-the-art models on benchmark datasets

Handles highly incomplete matrices effectively

Learns entry similarities during training

Abstract

The problem of completing high-dimensional matrices from a limited set of observations arises in many big data applications, especially, recommender systems. Existing matrix completion models generally follow either a memory- or a model-based approach, whereas, geometric matrix completion models combine the best from both approaches. Existing deep-learning-based geometric models yield good performance, but, in order to operate, they require a fixed structure graph capturing the relationships among the users and items. This graph is typically constructed by evaluating a pre-defined similarity metric on the available observations or by using side information, e.g., user profiles. In contrast, Markov-random-fields-based models do not require a fixed structure graph but rely on handcrafted features to make predictions. When no side information is available and the number of available observations becomes very low, existing solutions are pushed to their limits. In this paper, we propose a geometric matrix completion approach that addresses these challenges. We consider matrix completion as a structured prediction problem in a conditional random field (CRF), which is characterized by a maximum a posterior (MAP) inference, and we propose a deep model that predicts the missing entries by solving the MAP inference problem. The proposed model simultaneously learns the similarities among matrix entries, computes the CRF potentials, and solves the inference problem. Its training is performed in an end-to-end manner, with a method to supervise the learning of entry similarities. Comprehensive experiments demonstrate the superior performance of the proposed model compared to various state-of-the-art models on popular benchmark datasets and underline its superior capacity to deal with highly incomplete matrices.

Tables

Table 1. TABLE I : Descriptions of the datasets used in the experiments.

Dataset	# Users	# Items	# Ratings	Rating levels
MovieLens [53]	943	1,682	100,000	$1,2\dots 5$
Flixster [54]	3,000	3,000	26,173	$0.5,1\dots 5$
Douban [24]	3,000	3,000	136,891	$1,2\dots 5$
YahooMusic [55]	3,000	3,000	5,335	$1,2\dots 100$
Epinions [56]	40,163	139,738	664,824	$1,2\dots 5$

Table 2. TABLE II : The RMSE results of the DCMC model on a random split of the MovieLens dataset  [ 53 ] when varying β 𝛽 \beta .

$\beta$	$0.0$	$0.5$	$1.0$	$1.5$	$2.0$	$2.5$	$3.0$
RMSE	$0.896$	$0.892$	$0.892$	$0.891$	$0.892$	$0.893$	$0.893$

Table 3. TABLE III : The RMSE results of the DCMC model on a random split of the MovieLens dataset  [ 53 ] when varying γ 𝛾 \gamma .

$\gamma$	$0.01$	$0.03$	$0.05$	$0.075$	$0.1$	$0.25$	$0.5$
RMSE	$0.891$	$0.891$	$0.890$	$0.892$	$0.893$	$0.893$	$0.899$

Table 4. TABLE IV : The RMSE results of the DCMC model on a random split of the MovieLens dataset  [ 53 ] when varying T 𝑇 T .

$T$	$1$	$3$	$5$	$10$	$15$	$30$
RMSE	$0.892$	$0.891$	$0.890$	$0.890$	$0.890$	$0.890$

Table 5. TABLE V : The results on the MovieLens dataset  [ 53 ] for different models.

	RMSE	MAE
I-Autorec [18]	$0.905\pm 3e-4$	$0.712\pm 4e-4$
U-Autorec [18]	$0.980\pm 3e-4$	$0.781\pm 2e-4$
Deep U-Autorec [20]	$0.986\pm 3e-3$	$0.775\pm 1e-3$
m-I-Autorec [21]	$0.898\pm 4e-4$	$0.708\pm 3e-4$
m-U-Autorec [21]	$0.944\pm 4e-4$	$0.748\pm 3e-4$
NMC [22]	$0.905\pm 1e-3$	$0.716\pm 7e-4$
CF-NADE [58]	$0.901\pm 2e-3$	$0.696\pm 2e-3$
sRGCNN [33]	$0.933\pm 3e-3$	$0.738\pm 3e-3$
GCMC [34]	$0.908\pm 2e-3$	$0.712\pm 3e-3$
DCMC (Ours)	$0.893\pm \mathrm{𝟏}𝐞-\mathrm{𝟑}$	$0.694\pm \mathrm{𝟏}𝐞-\mathrm{𝟑}$

Table 6. TABLE VI : The results on the Flixster  [ 54 ] , Douban  [ 24 ] and YahooMusic  [ 55 ] datasets for different models.

Model	Flixster		Douban		YahooMusic
	RMSE	MAE	RMSE	MAE	RMSE	MAE
I-Autorec [18]	$0.884\pm \mathrm{𝟑}𝐞-\mathrm{𝟓}$	$0.652\pm \mathrm{𝟒}𝐞-\mathrm{𝟓}$	$0.807\pm 3e-3$	$0.638\pm 2e-3$	$21.663\pm 4e-2$	$16.143\pm 4e-2$
U-Autorec [18]	$1.135\pm 7e-6$	$0.899\pm 1e-5$	$0.785\pm 1e-4$	$0.622\pm 1e-4$	$26.622\pm 7e-2$	$21.384\pm 6e-2$
Deep U-Autorec [20]	$0.982\pm 8e-3$	$0.754\pm 7e-3$	$0.750\pm 2e-3$	$0.591\pm 2e-3$	$39.716\pm 1.070$	$30.474\pm 7e-1$
m-I-Autorec [21]	$0.940\pm 5e-4$	$0.723\pm 4e-4$	$0.750\pm 2e-4$	$0.585\pm 2e-4$	$32.541\pm 2e-2$	$24.679\pm 2e-2$
m-U-Autorec [21]	$0.998\pm 4e-4$	$0.708\pm 6e-4$	$0.758\pm 2e-4$	$0.598\pm 1e-4$	$38.762\pm 3e-2$	$29.414\pm 3e-2$
NMC [22]	$0.984\pm 3e-3$	$0.728\pm 2e-3$	$0.758\pm 1e-3$	$0.594\pm 1e-3$	$28.566\pm 1e-1$	$22.046\pm 7e-2$
CF-NADE [58]	$0.934\pm 3e-3$	$0.668\pm 2e-3$	$0.742\pm 1e-3$	$0.577\pm 1e-3$	N/A	N/A
sRGCNN [33]	$0.923\pm 7e-3$	$0.707\pm 7e-3$	$0.797\pm 3e-3$	$0.633\pm 2e-3$	$22.676\pm 6e-1$	$18.543\pm 4e-1$
GCMC [34]	$0.926\pm 3e-3$	$0.709\pm 4e-3$	$0.734\pm 5e-3$	$0.576\pm 7e-3$	$22.762\pm 7e-1$	$19.325\pm 8e-1$
DCMC (Ours)	$0.899\pm 2e-3$	$0.662\pm 1e-3$	$0.731\pm \mathrm{𝟓}𝐞-\mathrm{𝟒}$	$0.567\pm \mathrm{𝟕}𝐞-\mathrm{𝟒}$	$19.362\pm \mathrm{𝟗}𝐞-\mathrm{𝟐}$	$15.806\pm \mathrm{𝟕}𝐞-\mathrm{𝟐}$

Table 7. TABLE VII : The results on the Epinions dataset  [ 56 ] for different models.

Model	RMSE	MAE
I-Autorec [18]	$1.194\pm 2e-4$	$0.919\pm 1e-5$
U-Autorec [18]	$1.107\pm 3e-4$	$0.832\pm 5e-5$
Deep U-Autorec [20]	$1.204\pm 3e-2$	$0.881\pm 2e-2$
m-I-Autorec [21]	$1.182\pm 2e-3$	$0.858\pm 3e-3$
m-U-Autorec [21]	$1.113\pm 1e-3$	$0.845\pm 2e-3$
NMC [22]	$1.115\pm 8e-3$	$0.866\pm 9e-3$
GCMC [34]	$1.173\pm 2e-3$	$0.953\pm 4e-4$
DCMC (Ours)	$1.095\pm \mathrm{𝟖}𝐞-\mathrm{𝟒}$	$0.817\pm \mathrm{𝟑}𝐞-\mathrm{𝟑}$

Table 8. TABLE VIII : Comparing different train/test variants of the proposed model on the MovieLens dataset. MF stands for Mean-Field inference.

		Testing
		RMSE		MAE
		w/o MF	with MF	w/o MF	with MF
Training	w/o MF	$0.905$	$0.900$	$0.697$	$0.695$
	with MF	$0.913$	$0.893$	$0.702$	$0.694$

Equations

$$R^{\ast }=\arg \underset{R}{\min }\Vert {\mathcal{A}}_{\Omega }(R-M){\Vert }_F,$$

$${\mathrm{x}}^{\ast }=\arg \underset{\mathrm{x}}{\max }P(\mathrm{x}|\mathrm{O}),$$

$$P(\mathrm{x}|\mathrm{O})=\frac{1}{Z}\exp (-E(\mathrm{x})).$$

$$E(\mathrm{x})=\sum _{k\in \mathcal{V}}\Phi ({\mathrm{x}}_k^u)+\sum _{k,l\in \mathcal{N}}\Psi ({\mathrm{x}}_k^u,{\mathrm{x}}_l^v).$$

$$Q(\mathrm{x}|\mathrm{O})=\prod _{k\in \mathcal{V}}{Q}_k({\mathrm{x}}_k),$$

$$q_k^u=\frac{1}{Z_k}\exp \left\{-\left(\Phi ({\mathrm{x}}_k^u)+\sum _{l\in {\mathcal{N}}_k}\sum _{v=1}^p{q}_l^v\Psi ({\mathrm{x}}_k^u,{\mathrm{x}}_l^v)\right)\right\},$$

$$Z_k=\sum _{u=1}^p{q}_k^u$$

$$G_{ij}^u=U_i^{⊺}{B}^u{V}_j,$$

$$P(M_{ij}={\mathcal{L}}_u)=\frac{\exp (G_{ij}^u)}{{\sum }_{v=1}^p\exp (G_{ij}^v)}.$$

$$s^r(i_1,i_2)$$

$$s^c(j_1,j_2)$$

$$s\left(i_1{j}_1,i_2{j}_2\right)=s^r(i_1,i_2)\times s^c(j_1,j_2).$$

$$s(i_1{j}_1,i_2{j}_2)=\frac{U_{i_1}^{⊺}{U}_{i_2}}{\Vert U_{i_1}{\Vert }_2\Vert U_{i_2}{\Vert }_2}\times \frac{V_{j_1}^{⊺}{V}_{j_2}}{\Vert V_{j_1}{\Vert }_2\Vert V_{j_2}{\Vert }_2}.$$

$$\Phi ({\mathrm{x}}_k^u)=-\ln \left(P\left(M_{ij}={\mathcal{L}}_u\right)\right),$$

$$\Psi ({\mathrm{x}}_k^u,{\mathrm{x}}_l^v)=\gamma \times s(k,l)\times \mu (u,v),$$

$$\mu (u,v)=\min \left\{{\left({\mathcal{L}}_u-{\mathcal{L}}_v\right)}^2,\tau \right\},$$

$$q_k^u=\frac{1}{Z_k}\exp \left\{-\Phi ({\mathrm{x}}_k^u)-\sum _{l\in {\mathcal{N}}_k}\sum _{v=1}^p{q}_l^v\cdot \gamma \cdot s(k,l)\cdot \mu (u,v)\right\},$$

$$q_k^u=\frac{1}{Z_k}\exp \left\{-\Phi ({\mathrm{x}}_k^u)-\gamma \sum _{l\in {\mathcal{N}}_k}s(k,l)\sum _{v=1}^p{q}_l^v\cdot \mu (u,v)\right\}.$$

$${\mathcal{Q}}^{{}^{\prime }}=\mathrm{conv}(\mathcal{Q},\mathcal{C}),$$

$${\mathcal{Q}}_{k,u}^{{}^{\prime }}$$

$$=\sum _{v=1}^p{q}_k^v\cdot \mu (u,v).$$

$${\mathcal{Q}}_{k,u}^{{}^{\prime \prime }}=\sum _{l=1}^K{\mathcal{S}}_{k,l}\cdot {\mathcal{Q}}_{l,u}^{{}^{\prime }}.$$

$${\mathcal{Q}}_{k,u}^{{}^{\prime \prime }}=\sum _{l=1}^{K}s(k,l)\sum _{v=1}^p{q}_l^v\cdot \mu (u,v).$$

$$L_p=\frac{1}{c}\sum _{k=1}^c-\ln {\mathcal{P}}_{k,{\hat{R}}_k^t}^t,$$

$$s^{gt}(k,l)=\exp \left(\frac{-{\left(M_{i_1{j}_1}-M_{i_2{j}_2}\right)}^2}{{\sigma }^2}\right),$$

$$L_s=\frac{1}{|{\mathcal{N}}_c|}\sum _{k,l\in {\mathcal{N}}_c}{\left(s(k,l)-s^{gt}(k,l)\right)}^2,$$

$$L=L_p+\beta L_s,$$

$$R_k^{\ast }=\sum _{u=1}^p{\mathcal{L}}_u\cdot {\mathcal{P}}_{k,u}.$$

$$=\sqrt{\sum _{ij\in {\Omega }_{\text{test}}}(R_{ij}-M_{ij}{)}^2/\left|{\Omega }_{\text{test}}\right|},$$

$$=\sum _{ij\in {\Omega }_{\text{test}}}|R_{ij}-M_{ij}|/\left|{\Omega }_{\text{test}}\right|,$$

Full text

I Introduction

MATRIX completion is a fundamental problem in machine learning and signal processing, with a wide range of applications spanning from recommender systems [1, 2] to image inpainting [3, 4]. The problem is defined as follows: given a partially observed matrix $M\in\mathbb{R}^{n\times m}$ with $\Omega$ the set of indices of known entries, recover the unknown entries in $M$. This can be achieved by solving for a dense prediction matrix $R^{*}\in\mathbb{R}^{n\times m}$ such that:

[TABLE]

with $\mathcal{A}_{\Omega}$ an operator that selects the entries defined in $\Omega$, and $\|\cdot\|_{F}$ the Frobenius norm. Among the most notable applications of matrix completion is collaborative filtering in recommender systems, where $M$ is the rating matrix with its rows and columns corresponding to users and items, and an entry representing the rating or interaction between a user and an item. Only a small number of entries are typically observed since an average user only rates a small portion of the items. Due to this scarcity of observed entries, predicting the missing entries becomes highly challenging.

Most existing matrix completion methods follow either a memory- or a model-based approach [5]. Memory-based, alias, $k$-nearest-neighbors ($k$-NN), methods predict the missing ratings by utilizing the relationships among the users and/or items: a missing rating of a user for an item is predicted by using the known ratings of similar users for the same item (user-based method) or the known ratings of the same user for similar items (item-based method) [5, 6, 7]. A critical step in such methods is the estimation of how similar users or items are. This similarity is often estimated by evaluating pre-defined metrics—such as the cosine similarity or the Pearson correlation [6]—on the common known entries. Hybrid memory-based methods fuse the user- and item-based views [8, 9, 10], leading to more reliable predictions [9]. Memory-based models, in general, rely on a subset of the available information [5], and can be unreliable due to the data scarcity problem [9].

Model-based methods, on the other hand, predict the missing entries by regularizing Problem (1) using functions that impose underlying low-complexity characteristics on the data, i.e., low-rank [11, 12, 13], low-rank-plus-sparse [14, 15], or non-negativity characteristics [16, 17]. By solving the regularized optimization problems, such models learn latent representations from the data and often provide more accurate predictions compared to memory-based methods [5]. Recently, deep-learning-based methods, such as deep autoencoders [18, 19, 20, 21] and deep matrix factorization models [22, 23], learn non-linear latent representations, leading to high performance. However, the low-complexity characteristics imposed by such approaches might not be present or might underfit the underlying structure in the data, resulting in performance loss.

Various methods have been proposed to combine the best of both the memory- and model-based approaches. In this direction, geometric matrix completion (GMC) has lately received a lot of attention [24, 25, 26]. GMC refers to model-based methods that leverage the relationships among the set of users and items when making predictions [24, 25, 27, 28]. These relationships are often represented in the form of graphs, called structure graphs, with nodes representing users, items or entries, and edges encoding their similarities. Several methods proposed to leverage Markov random fields (MRF) to encode the relationships between the matrix entries [29, 30, 31, 32]. In these methods, the structure graphs were incorporated into the estimation of the MRF potentials. MRF-based methods can be highly flexible, in the sense that they can learn the structure graphs directly from the data and do not require specifying the edge weights beforehand [31]. Nevertheless, they rely on handcrafted features for estimating the potentials. Recently, several methods have been proposed to leverage graph deep learning techniques to learn the features from the data while utilizing the structure graphs [33, 34, 35]. These methods have achieved promising performance on various benchmark datasets [33, 34, 35]. However, they require fully-defined user and item graphs to operate on. The user and item graphs were often built using pre-defined similarity metrics [27], which become unreliable in case of high data scarcity [9], or using side information [33, 35], which is not always available.

In this paper, we focus on the challenge of completing matrices from very few observations, without assuming access to side information (for example, user or item profiles). We propose a geometric matrix completion model that (i) leverages a deep neural network architecture to learn the latent representations in the data, (ii) learns the structure graphs and the relationships between entries directly from the data. We consider matrix completion as a structured prediction problem in a conditional random field (CRF), which is characterized by a maximum a posteriori (MAP) inference problem. We employ the mean-field algorithm to approximately solve this MAP inference, and propose a mechanism to unfold the algorithm into neural network layers. This unfolding mechanism allows us to incorporate the mean-field inference on top of a deep neural network, resulting in the proposed deep conditional random fields model for matrix completion (DCMC). As such, the proposed model simultaneously carries the advantages of different state-of-the-art approaches: it learns the latent features in the data (advantage of deep learning); it learns the structure graph from the data (advantage of MRF-based matrix completion models).

Our main contributions are as follows:

• •

We propose a deep CRF model for matrix completion, which simultaneously computes the CRF potentials, estimates the relationships between entries and performs mean-field inference in each forward pass. The proposed model can be trained end-to-end using only the known matrix entries.

• •

We propose a method to supervise the learning of the similarities between entries by utilizing the known matrix entries. Using this method the model effectively learns the structure graphs from the available data.

• •

We perform comprehensive experiments on well-established benchmark datasets, which demonstrate (i) the gain in prediction accuracy that the proposed DCMC model brings over various state-of-the-art models, and (ii) the effectiveness of the learned similarities compared to those estimated using pre-defined metrics. The results corroborate that the improvements are more profound on datasets with very few observed entries.

The remainder of the paper is organized as follows. In Section II, we review the related work, and in Section III, we present our formulation of matrix completion as a MAP inference problem in CRF. In Section IV, we describe our deep geometric matrix completion model, and present the experimental settings and results in Section V. Finally, we draw the conclusion in Section VI.

II Related Work

II-A Hybrid Memory-based Matrix Completion

Memory-based, or $k$-NN, methods predict a missing rating $M_{ij}$ by (weighted) averaging the values of the $k$ entries most similar to it from a set of potential predicting entries. User- and item-based $k$-NN methods consider potential predicting entries along the $i$-th row and the $j$-th column of the matrix $M$. An illustration of the relationships between matrix entries is given in Fig. 1.

With limited known entries, fewer potential predicting entries are available; as a result, the predictions of $k$-NN methods become unreliable [9]. To mitigate this problem, hybrid memory-based methods unify the user- and item-based views to enrich the set of potential predicting ratings [9, 8]. In [9], for example, in order to predict a missing rating $M_{ij}$, the ratings of user $i$ for the other items (ratings along the $i$-th row), the ratings of the other users for item $j$ (ratings along the $j$-th column), and the ratings of other users for other items are considered.

In this work, we follow a model-based approach and propose a deep neural network which simultaneously (i) finds the latent factors in the data, and (ii) learns and leverages the relationships among the matrix entries. Our model incorporates the advantages of the hybrid-memory-based methods [9, 8] as it considers all entries as potential predicting entries when making predictions; yet, it does not rely on pre-defined similarity metrics (e.g., the Pearson correlation), which becomes unreliable in case of high data scarcity [9].

II-B Geometric Matrix Completion

Geometric matrix completion (GMC) methods incorporate structure graphs, which capture relationships between users, items or entries, into the prediction model. Structure graphs were used to regularize prediction models by enforcing a smoothness constraint on the latent user factors [24], on the rows and columns of the dense prediction matrix [27] or on both the latent user and item factors [28] via graph regularization techniques.

With the goal to exploit the structure graphs, several GMC methods utilize random field models, which are powerful in modeling the dependencies between random variables. The Preference Network [29], for example, used a Markov random fields (MRF) model with nodes representing entries and edges encoding their relationships. This model was later extended to handle ordinal rating values [30] and relative preferences, i.e., item rankings [32]. Alternatively, the item field model [36] built an MRF model on top of the item graphs. These models, however, require the structure graphs to be defined before constructing the models. Tran et al. alleviated this requirement and proposed an MRF-based model in which the edges in the graphs were parameterized and learned from the data [31]. Nevertheless, these MRF-based methods rely on handcrafted features to compute the potentials of the random fields.

More recent methods leverage geometric deep learning techniques [37] to learn the latent features using the structure graphs. Monti et al. proposed a multi-graph convolutional neural network to capture the spatial features from the user and item graphs [33], which were built using side information (e.g., information obtained from user profiles). Berg et al. employed a bipartite graph, constructed from the original matrix, with nodes corresponding to users and items, and edges corresponding to the known ratings. They proposed to use a convolutional graph encoder to encode the nodes into latent factors [34]. Wu et al. constructed the user and item graphs from side information (similar to [33]) and employed graph convolutional neural networks [38] to learn the latent factors. Geometric-deep-learning-based methods have achieved high performance on several benchmark datasets [33, 34, 35]; nevertheless, they require the structure graphs to be fully-specified before training.

Our model is classified into the GMC category of models, especially, the deep-learning-based ones. Unlike existing deep-learning-based GMC models, our model learns the structure graph from the known matrix entries and incorporates them in a conditional random field. Unlike existing random-field-based models, our model learns the latent features of the data by leveraging a deep neural network architecture. Furthermore, it utilizes all relationships among all available entries, instead of using only the relationships between entries that share common users or items (as MRF-based [31] methods do).

II-C Deep Random Fields Models

Random field models have been successfully applied to solve various problems in natural language processing (NLP) [39, 40] and computer vision [41, 42]. Traditional methods followed a two-stage pipeline in which the random fields models are used in a separate post-processing stage to enforce smoothness over the outputs of dependent nodes, given the potentials estimated at the first stage. Recent studies in the computer vision domain have shown that combining random fields and deep neural networks into a joint model can significantly boost the performance [43, 44, 45].

We draw inspiration from these models, and build a deep Conditional random field (CRF) model for matrix completion by unfolding the inference step in the CRF into neural network layers. An inherent challenge that appears when applying this stategy in matrix completion is the lack of an explicit local neighborhood between matrix entries, as opposed to the neighborhood of pixels in visual data (e.g., the widely-used 4-connected or 8-connected neighborhoods in images and videos). To overcome this challenge, we adopt a fully-connected CRF where a node is connected to all the other nodes. This full connectivity requires us to develop a new unfolding mechanism for the inference in the CRF, and methods to learn reliable relationships among the nodes. To the best of our knowledge, this is the first work to successfully solve the matrix completion problem with a deep random field model.

III Matrix Completion as Structured Prediction in CRF

In this section, we formulate matrix completion as a MAP inference problem in a CRF and then describe the mean-field algorithm that can solve the specific inference problem.

Suppose for now that we have obtained the relationships between matrix entries. We will describe in detail how we learn these relationships in Section IV-A2. Let us consider a CRF defined over an undirected graph $\mathcal{G}=(\mathcal{V},\mathcal{E})$, where $\mathcal{V}$ is the set of nodes, with each node corresponding to an entry in the matrix $M$, and $\mathcal{E}$ is the set of edges whose weights encode the relationships between the nodes. In matrix completion, there is no explicit local neighborhood for an entry. For this reason, we opt to encode all the pairwise relationships between the nodes by making $\mathcal{G}$ fully-connected. A downside of constructing $\mathcal{G}$ as a graph of entries is that the number of nodes becomes very large in applications involving high-dimensional matrices. We introduce several techniques to alleviate this problem within our model in Section IV-E and Section IV-F.

Denote by $K$ the number of nodes in $\mathcal{G}$, i.e., $|\mathcal{V}|$, we have $|\mathcal{E}|=K^{2}$. A node $k$ in the CRF is associated with a latent random variable $X_{k}$ representing the label (alias, the value) of the corresponding entry. The random variables $X_{k}$, $k=1,\dots,K$, have domain $\mathcal{L}$. We consider discrete matrices, e.g., rating matrices, hence, $\mathcal{L}=\{\mathcal{L}_{1},\mathcal{L}_{2},\dots,\mathcal{L}_{p}\}$, with $p$ the number of possible entry values. The edge between the nodes $k$ and $l$ encodes the statistical dependency between the random variables $X_{k}$ and $X_{l}$. In the rest of the paper, we refer to the nodes and the labels by their indices, namely $k,l\in\{1,\dots,K\}$ and $u,v\in\{1,\dots,p\}$.

We denote by $\mathrm{O}=\mathcal{A}_{\Omega}(M)$ the observations over the matrix $M$, i.e., the given entries, and by $\mathrm{x}\in\mathcal{L}^{K}$ a labeling operator that assigns to each node in $\mathcal{G}$ a label in $\mathcal{L}$. Each instantiation of $\mathrm{x}$ indicates a sequence of labels for the CRF’s nodes. By taking the labels for the missing entries from an instantiation of $\mathrm{x}$, one can complete the matrix $M$. Finding the best predictions for the missing entries’ values is, therefore, equivalent to finding the most probable instantiation of $\mathrm{x}$ given $\mathrm{O}$. This procedure can be formulated as a MAP inference problem:

[TABLE]

with $P(\mathrm{x}|\mathrm{O})$ the posterior in the CRF, which is given by:

[TABLE]

In (3), $Z$ is the partition function ensuring a valid distribution, and $E(\mathrm{x})$ is the energy of the CRF, which has the form:

[TABLE]

The unary potential $\Phi(\mathrm{x}_{k}^{u})$ in (4) measures the cost of assigning the label $\mathcal{L}_{u}$ to the node $k$. This cost is computed for each node and each label. The computation of the unary potential can be done in a separate step before the CRF inference, e.g., by means of a prediction model. The pairwise potential $\Psi(\mathrm{x}_{k}^{u},\mathrm{x}_{l}^{v})$ measures the cost of assigning to nodes $k$ and $l$, the labels $\mathcal{L}_{u}$ and $\mathcal{L}_{v}$, respectively. $\mathcal{N}$ is the set of all connected pairs in the CRF; in our model, $|\mathcal{N}|=|\mathcal{E}|=K^{2}$. Intuitively, $\Psi(\mathrm{x}_{k}^{u},\mathrm{x}_{l}^{v})$ encodes the relationship between the two corresponding entries. Unlike existing MRF-based models for matrix completion (e.g., [31]), where the pairwise potentials were only computed for pairs of entries of the same users or the same items, the pairwise potentials in our model are computed for all pairs of matrix entries. Furthermore, as shown in Section IV-A, both the unary and pairwise potentials of our CRF are computed using a deep neural network.

As exactly computing the posterior $P(\mathrm{x}|\mathrm{O})$ is intractable, we employ the mean-field algorithm to approximate the posterior $P(\mathrm{x}|\mathrm{O})$ [46]. In what follows, we briefly describe this algorithm, the steps of which are interpreted as neural network layers within our model (see Section IV-C). The mean-field algorithm approximates $P(\mathrm{x}|\mathrm{O})$ by a simpler proposal distribution $Q(\mathrm{x}|\mathrm{O})$ belonging to the family of fully-factorized distributions:

[TABLE]

where $Q_{k}(\mathrm{x}_{k})$ is the distribution over the variable $X_{k}$ and $Q_{k}(\mathrm{x}_{k}^{u})$ is the probability of labeling the node $k$ with the label $\mathcal{L}_{u}$ according to the distribution $Q_{k}$. Then, the algorithm [46] tries to find the proposal distribution $Q$ that is as close as possible to the target distribution $P$, where the closeness is measured via the Kullback-Leibler divergence $\mathbf{D}_{\text{KL}}(Q||P)$. For brevity, let us denote $Q_{k}(\mathrm{x}_{k}^{u})$ as $q_{k}^{u}$; the mean-field algorithm [46] estimates $q_{k}^{u}$, for all $k\in\{1,\dots,K\}$ and $u\in\{1,\dots,p\}$, by minimizing $\mathbf{D}_{\text{KL}}(Q||P)$ with respect to each $q_{k}^{u}$, subject to the constraint $\sum_{u=1}^{p}q_{k}^{u}=1,\forall k\in\{1,\dots,K\}$. This is done by means of the following generic mean-field update equation [46]:

[TABLE]

with $\mathcal{N}_{k}$ the set of nodes connected to the node $k$, and $Z_{k}$ the normalization factor to make $Q_{k}$ a valid probability distribution:

[TABLE]

The mean-field algorithm iteratively updates $q_{k}^{u}$ according to (6), for all $k\in\{1,\dots,K\},u\in\{1,\dots,p\}$, for a certain number of iterations, or until a convergence condition has been reached, e.g., the changes in all $q_{k}^{u}$ fall below a small tolerance value. The result is the proposal distribution $Q^{*}$ that best approximates $P(\mathrm{x}|\mathrm{O})$. As $Q^{*}$ is fully-factorized, the solution to the MAP problem (2) can be found by taking for each node $k$ the label that maximizes the marginal distribution $Q^{*}_{k}$.

IV Deep CRF for Matrix Completion

In this section, we first describe our deep neural network that simultaneously estimates the similarities between entries and computes the unary and pairwise potentials of the CRF. We refer to this neural network as the base prediction network. Using the computed potentials, we derive our final mean-field update equation. After that, we present a method to perform the mean-field update using specially-designed neural network layers, which we call mean-field layers. Stacking these mean-field layers on top of the base prediction network forms our Deep CRF model for matrix completion (DCMC). The architecture of the DCMC model is illustrated in Fig. 2. At the end of the section, we present methods to efficiently train and make predictions with the proposed model, and to effectively supervise the learning of the similarity between entries.

IV-A The Base Prediction Network

The architecture of the base prediction network is depicted in blue in Fig. 2. This architecture is inspired by our previous deep matrix factorization models in [22, 23]. The base prediction network has two branches, called the row and column branches, which consist of a configurable number of fully connected layers, each followed by a batch normalization layer [47]. All layers, except for the last ones in each branch, are followed by the Rectified Linear Unit (ReLU) activation function [48] and dropout regularization [49]. The network takes as inputs a batch of row vectors ($M^{r}$) and a batch of column vectors ($M^{c}$) from the original matrix $M\in\mathbb{R}^{n\times m}$. Similar to [18, 22], we impute missing entries with [math]. The row and column branches transform these input vectors into embeddings in the $d$-dimensional latent space: Given a row vector $M^{r}_{i}\in\mathbb{R}^{m}$ and a column vector $M^{c}_{j}\in\mathbb{R}^{n}$, the two branches produces two embeddings $U_{i},V_{j}\in\mathbb{R}^{d\times 1}$, respectively. Using these embeddings, the score $G_{ij}^{u}$ for the entry at position $(i,j)$ and label $\mathcal{L}_{u}$, is calculated via a bi-linear decoder:

[TABLE]

with $B^{u}\in\mathbb{R}^{d\times d}$ learnable weights for the label $\mathcal{L}_{u}\in\mathcal{L}$. As there are $p$ labels in $\mathcal{L}$, the bi-linear decoder consists of $p\times d\times d$ parameters in total.

IV-A1 Label Probability

Using the predicted scores $G_{ij}^{u}$, the predicted probability $P(M_{ij}=\mathcal{L}_{u})$ is computed using the softmax function:

[TABLE]

IV-A2 Computing the Entry Similarity

It is worth noting that we focus on the cases where at least one of the two entries are unknown, since calculating the similarity between two known entries is trivial. We denote by $s^{r}$ and $s^{c}$, respectively, the functions that compute the user and item similarities: $s^{r}(i_{1},i_{2})$ computes the similarity between users $i_{1}$ and $i_{2}$, and $s^{c}(j_{1},j_{2})$ computes the similarity between items $j_{1}$ and $j_{2}$. As the cosine similarity has been proven effective and robust in measuring similarities between high dimensional vectors in learned latent spaces [50], we define $s^{r}$ and $s^{c}$ as the cosine similarity between the embeddings produced by the base prediction network, namely,

[TABLE]

with $U_{i}$, $V_{j}$ the embeddings of user $i$ and item $j$.

We model the similarity between two entries as the product of the corresponding user and item similarities. With the assumption that $s^{r}$ and $s^{c}$ are non-zero, if two users have similar preferences [that is, $s^{r}(i_{1},i_{2})$ is high], their ratings for similar items [that is, $s^{c}(j_{1},j_{2})$ is high] should be similar. Whereas, if two users have dissimilar preferences [that is, $s^{r}(i_{1},i_{2})$ is low], they are not expected to have similar ratings. Denote by $s$ the function that computes the entry similarity, the similarity between two matrix entries $M_{i_{1}j_{1}}$ and $M_{i_{2}j_{2}}$, $s\left(i_{1}j_{1},i_{2}j_{2}\right)$, is given by

[TABLE]

Using $s^{r}$ and $s^{c}$, which are defined in (10), the entry similarity is computed as

[TABLE]

As the cosine similarity has a range of $\left[-1,+1\right]$, we linearly scale $s^{r}(i_{1},i_{2})$ and $s^{c}(j_{1},j_{2})$ so that they lie in $\left[0,1\right]$. The entry similarity, then, is also in the range $\left[0,1\right]$.

IV-B Modeling the Unary and Pairwise Term

We now present how we compute the unary and pairwise potentials using the outputs of the base prediction network.

IV-B1 The Unary Potentials

The unary potential $\Phi(\mathrm{x}_{k}^{u})$ measures the cost of assigning the label $\mathcal{L}_{u}$ to a node $k$. We use the negative log-likehood to compute $\Phi(\mathrm{x}_{k}^{u})$. $\Phi(\mathrm{x}_{k}^{u})$ will be high if for the node $k$ the label $\mathcal{L}_{u}$ has low score and vice versa. Specifically, suppose that the node $k$ corresponds to the entry $M_{ij}$, then, the unary term $\Phi(\mathrm{x}_{k}^{u})$ is computed as

[TABLE]

where $P\left(M_{ij}=\mathcal{L}_{u}\right)$ is the predicted label probability that is computed using (9).

IV-B2 The Pairwise Potentials

The pairwise potentials measure the label disagreement cost between pairs of nodes in the model. We compute the cost of assigning the labels $\mathcal{L}_{u}$ and $\mathcal{L}_{v}$ to the nodes $k$ and $l$ as

[TABLE]

with $\gamma$ a hyperparameter determining the weight of the pairwise term with respect to the unary term and $s(k,l)$ the estimated similarity between the nodes $k$ and $l$. Here, the nodes $k,l$ respectively correspond to the entries at the positions $(i_{1},j_{1})$ and $(i_{2},j_{2})$ in the matrix $M$, and the similarity $s(k,l)$ is estimated according to (12). In (14), $\mu(u,v)$ is a function that computes the compatibility between the labels $\mathcal{L}_{u},\mathcal{L}_{v}$, which is often referred to as the compatibility function in the random fields literature. There are many forms of $\mu$ that have been used for CRF models; in this work, we employ the truncated quadratic function:

[TABLE]

with $\tau$ a pre-defined truncation threshold.

It can be seen from (14) that the pairwise potentials $\Psi(\mathrm{x}_{k}^{u},\mathrm{x}_{l}^{v})$ depend on the learned entry similarity; as such, in our model, both the unary and pairwise potentials are computed from the learned latent features for the users and items, which are produced by the base prediction network.

IV-B3 The Final Mean-field Update

Substituting the unary and pairwise potentials in (13) and (14) into (6), we derive the final mean-field update equation for our model as

[TABLE]

or equivalently:

[TABLE]

We refer to the term $\sum_{v=1}^{p}q_{l}^{v}\cdot\mu(u,v)$ in (17) as the compatibility transform, and to the outer term $\sum_{l\in\mathcal{N}_{k}}s(k,l)\sum_{v=1}^{p}q_{l}^{v}\cdot\mu(u,v)$, which involves the summation over all the nodes connected to the node $k$, as the message passing operation.

IV-C Unfolding the Mean-field Algorithm

Let us suppose for now that we process all the $K$ entries in the matrix simultaneously in a full batch, namely, we use all the rows and columns of the given matrix $M\in\mathbb{R}^{n\times m}$ as the inputs to the base prediction network. The outputs of the base prediction network then consist of: (i) the label probability matrix, denoted by $\mathcal{Q}\in\mathbb{R}^{K\times p}$, with $\mathcal{Q}_{k,u}=q_{k}^{u}=P(M_{ij}=\mathcal{L}_{u}),\forall k\in\{1,\dots,K\},u\in\{1,\dots,p\}$, and $(i,j)$ is the location of the entry corresponding to node $k$; (ii) the learned entry similarity matrix $\mathcal{S}\in\mathbb{R}^{K\times K}$ of which each element $\mathcal{S}_{k,l}$ is the predicted similarity between the corresponding entries of nodes $k$ and $l$, $s(k,l)$; and (iii) the matrix of the unary terms $\Phi\in\mathbb{R}^{K\times p}$. Since we build a fully-connected CRF model, $\mathcal{S}$ is a dense matrix. We denote by $\mathcal{C}\in\mathbb{R}^{p\times p}$ the label compatibility matrix, each element $\mathcal{C}_{u,v}$ of which corresponds to the compatibility $\mu(u,v)$ between two labels $\mathcal{L}_{u},\mathcal{L}_{v}\in\mathcal{L}$. The matrix $\mathcal{C}$ can be calculated offline according to (15) on the possible entry values. In what follows, we describe how we unfold the mean field update, taking the matrices $\mathcal{Q}$, $\mathcal{S}$, $\Phi$, and $\mathcal{C}$ as input.

IV-C1 The Compatibility Transform Step

The compatibility transform can be performed via a 1-D convolutional layer applied on the matrix $\mathcal{Q}$. This convolutional layer has $p$ filters of kernel size $1\times p$ whose weights are determined from $\mathcal{C}$ as follows: the weights of the $u$-th filter are fixed equal to the values along the $u$-th row of $\mathcal{C}$. We do not employ any padding and set the stride to $1$. The $u$-th filter slides vertically across $\mathcal{Q}$, and calculates the inner product between its weights and the rows of $\mathcal{Q}$. The output of this layer, which is denoted as $\mathcal{Q}^{{}^{\prime}}\in\mathbb{R}^{K\times p}$, is given by

[TABLE]

where $\operatorname{conv}(\mathcal{Q},\mathcal{C})$ denotes the operation of a convolutional layer on the input $\mathcal{Q}$ with filters constructed from $\mathcal{C}$ as described above. An element $\mathcal{Q}^{{}^{\prime}}_{k,u}$ with $k\in\{1,\dots,K\}$ and $u\in\{1,\dots,p\}$ is expressed as

[TABLE]

An illustration of this operation is given in Fig. 3(a), where the $u$-th and $v$-th column of $\mathcal{Q}^{{}^{\prime}}$ contains the results obtained by applying the 1-D convolution using the $u$-th and $v$-th filter, respectively, on $\mathcal{Q}$. In Fig. 3(a), these columns are displayed with the same colors as their corresponding filters.

IV-C2 The Message Passing Step

After multiplying $\mathcal{S}\in\mathbb{R}^{K\times K}$ with $\mathcal{Q}^{{}^{\prime}}\in\mathbb{R}^{K\times p}$, we get $\mathcal{Q}^{{}^{\prime\prime}}\in\mathbb{R}^{K\times p}$, where an element $\mathcal{Q}^{{}^{\prime\prime}}_{k,u}$ is given by

[TABLE]

After expanding $\mathcal{Q}^{{}^{\prime}}_{l,k}$ according to (19), (20) becomes

[TABLE]

An illustration of this operation is given in Fig. 3(b). As our graph of entries is fully-connected, the set of nodes connected to the node $k$ is given by $\mathcal{N}_{k}=\{1,\dots,K\}$. Therefore, $\mathcal{Q}^{{}^{\prime\prime}}_{k,u}$ is the result of the message passing step in (17).

IV-C3 The Mean-field Layer

After the compability transform and message passing steps, the remaining operations involved in one mean-field iteration can be performed straightforwardly. Algorithm 1 summarizes one iteration of the unfolded mean-field update. The step that adds the unary potentials involves element-wise products and element-wise additions, and the update and normalization step can be performed simultaneously for all the nodes and labels using the softmax function. We can group all operations in a mean-field iteration and consider them as a specially-designed neural network layer, called mean-field layer.

IV-D The DCMC Model

Using the techniques presented in Section IV-C1 and Section IV-C2, we can then interpret $T$ iterations of the mean-field algorithm into $T$ mean-field layers stacked on top of each other; namely, a subsequent layer takes the output $\mathcal{Q}$ from its preceding layer as input. All mean-field layers share the same set of parameters, that is $\mathcal{S},\Phi,\mathcal{C}$. This stack of mean-field layers (illustrated in green in Fig. 2) can then be put on top of the base prediction network (illustrated in blue in Fig. 2), forming our deep CRF model for matrix completion (DCMC). Each forward pass of the model involves computing entry similarities, estimating the CRF potentials and performing the mean-field updates. As all operations in a mean-field layer are differentiable, we can back-propagate the gradients of the loss function through each mean-field layer. This allows us to train the DCMC model using gradient descent algorithms in an end-to-end manner. It is worth noting that a mean-field layer does not introduce any additional free parameters to the model, hence, it does not increase the risk of overfitting of the final model.

Integrating the mean-field update on top of the prediction network allows training the prediction network with feedback from the mean-field layers. Intuitively, this allows the prediction network to learn to adapt to the mean-field inference. This is an advantage of the proposed model compared to using a two-stage method, which first performs the base prediction network to compute the potentials and then applies the mean-field algorithm.

IV-E Training the DCMC Model

So far, we have assumed working on the whole CRF model with $K$ nodes. Nevertheless, in applications involving big matrices, this becomes impractical due to the high computation and memory consumptions. We employ two techniques to mitigate this problem: (i) during training, we consider only the known entries as nodes in the CRF instead of all the matrix entries; and (ii) we train our model in mini-batches.

In a training iteration $t$, we randomly sample $n^{t}$ rows and $m^{t}$ columns from the original matrix. When evaluating the loss function, we only take into account the observed entries among all the sampled $n^{t}\times m^{t}$ entries. We denote this set of observed entries by $\hat{R}^{t}$, with $|\hat{R}^{t}|=c$. It should be noted that $c$ is different in each mini-batch. Similarly, we only consider the nodes corresponding to the $c$ observed entries in $\hat{R}^{t}$ when constructing the graph for the CRF. Implementation-wise, from the probability matrix $\mathcal{Q}^{t}\in\mathbb{R}^{K^{t}\times p}$, the matrix of unary terms $\Phi^{t}\in\mathbb{R}^{K^{t}\times p}$, and the similarity matrix $\mathcal{S}^{t}\in\mathbb{R}^{K^{t}\times K^{t}}$ produced by the base prediction network, we select sub-matrices $\tilde{\mathcal{Q}}^{t}\in\mathbb{R}^{c\times p}$, $\tilde{\Phi}^{t}\in\mathbb{R}^{c\times p}$ and $\tilde{\mathcal{S}}^{t}\in\mathbb{R}^{c\times c}$ using the indices of the observed entries. $\tilde{\mathcal{Q}}^{t}$ is used as the input to the first mean-field layer, while $\tilde{\Phi}^{t}$ and $\tilde{\mathcal{S}}^{t}$ are shared among all the mean-field layers.

Due to the mini-batch sampling, an entry only gets connected to other entries in the same mini-batch; hence, not all the relationships among the entries are utilized. To remedy this problem, we sample the row and column vectors according to an ordering and randomly shuffle this ordering after each epoch. By training for long enough, we expect to cover most of the relationships among the entries. In our experiments, we empirically observed that sampling $\mathcal{Q},\mathcal{C}$ and $\mathcal{S}$ during training does not affect the performance of the model.

Loss Function

We employ the cross entropy loss to train the DCMC model, which is calculated as

[TABLE]

with $\mathcal{P}^{t}$ the final probability matrix after the last mean field layer, and $\mathcal{P}^{t}_{k,\hat{R}^{t}_{k}}$ the probability of assigning to the node $k$ its ground-truth label.

Supervising the Similarity Learning

Given two entries with known values, we can straightforwardly calculate their similarity, which can be used as ground-truth data to supervise the similarity learning. We employ the Gaussian similarity function [51] to obtain the ground-truth similarities between the entries. This function is bounded in the range $[0,1]$, which is desired by our similarity modeling in (12). The ground-truth similarity between the nodes $k$ and $l$, which correspond to the entries $M_{i_{1}j_{1}}$ and $M_{i_{2}j_{2}}$, is calculated by

[TABLE]

where $\sigma^{2}$ is a hyperparameter. We use a loss term $L_{s}$ measuring the mean squared-error between the predicted and the ground-truth node similarities:

[TABLE]

with $\mathcal{N}_{c}$ the set of connections between two observed entries in each mini-batch. Applying this loss term on two entries of similar values will push the embeddings of the corresponding users and items to be close in the latent space, and pull their embeddings far apart otherwise. By applying the same loss on all pairs of observed entries the model is expected to produce embeddings that minimize the similarity loss globally. We empirically observe that supervising the similarity learning systematically improves the quality of the learned similarities, and boosts the performance of the DCMC model.

Our final loss function is then a weighted combination of the cross entropy and similarity losses:

[TABLE]

with $\beta$ a parameter balancing the two loss terms. The loss function in (25) is optimized over the model’s parameters using stochatistic gradient descent (SGD) algorithm with Adam parameter update [52].

IV-F Testing the DCMC Model

At the testing phase, a CRF model with nodes corresponding to all the $K$ entries in the given matrix $M\in\mathbb{R}^{n\times m}$ is constructed. After a forward pass of the model, we get the probability matrix $\mathcal{P}\in\mathbb{R}^{K\times p}$ where $\mathcal{P}_{k,u}$ is the probability of assigning a label $\mathcal{L}_{u}$ to node $k$. The continuous prediction $R^{*}_{k}$ is given by

[TABLE]

When dealing with matrices of high dimensions, we randomly divide its rows and columns into subsets and perform predictions according to (26) inside each subset separately in order to reduce the computation and memory requirements. This procedure can be performed many times to produce multiple predictions for an entry, each time considering a different random set of predicting entries. The final entry value prediction can then be given by calculating their average.

V Experiments

In this section, we present our experimental studies. We first explain our experimental settings and the hyperparameter sensitivity of the DCMC model. After that, we compare the DCMC model against state-of-the-art deep-learning-based matrix completion models. Finally, we carry out experiments to justify the benefits of each component in the proposed model.

V-A Experimental Settings

Five real-world datasets are employed in our experiments, namely, the MovieLens [53], Flixster [54], Douban [24], YahooMusic [55] and Epinions [56] datasets. These datasets vary in the number of users and items, rating levels and context (movie, music and general consumer ratings). For the first four datasets, we use the experimental configurations (including train/test splits) provided by [33]. Regarding the Epinions dataset, we randomly split the known ratings into $75\%$ for training, $5\%$ for validation and $20\%$ for testing. The details of the five datasets are given in Table I. It can be seen that the densities of the observed entries vary across the datasets, from $6.30\%$ and $1.52\%$, respectively, on the MovieLens and Douban datasets to very low on the Flixster, YahooMusic and Epinion datasets ($0.29\%$, $0.06\%$ and $0.01\%$, respectively). It is worth mentioning that we do not employ any side information, e.g., user or item features, in our experiments.

We compare the DCMC model with state-of-the-art deep-learning-based matrix completion models. The performance of the models is assessed using the Root Mean Square Error (RMSE) and Mean Absolute Error (MAE),

[TABLE]

calculated over the entries reserved for testing (indexed by $\Omega_{\text{test}}$). Smaller RMSE and MAE values indicate more accurate predictions.

V-B Hyperparameters Selection

There are a number of hyperparameters that are related to the base prediction network, the mean-field layers, and the training stage. For the base prediction network, we follow [22, 23], and use 2 hidden layers in both the row and column branches. The number of hidden units in the first and second layers are set to 512 and 128, respectively. As the known entries available for training in the employed datasets are scarce, we use a high dropout rate of $0.75$ to mitigate overfitting. We empirically adapt some hyperparameters related to the mean-field layers to each dataset. Specifically, we set the truncation threshold $\tau$ for the quadratic compatibility function in (15) to $100.0$ on the YahooMusic dataset (values in the range $[1,100]$), and to $12.0$ on the other datasets (values in the range $[1,5]$). The value of $\sigma^{2}$ used to calculate the ground-truth entries’ similarities in (23) is set to $3000.0$ on the YahooMusic dataset and to $3.5$ on the other datasets. We set the number of training epochs to $300$ (we count one epoch each time all the rows or all the columns are sampled for training), and the learning rate to $0.01$ initially which is reduced by a factor of $0.5$ every $25$ epochs.

We determine the values of the following hyperparameters by cross validation, specifically: $\beta$, which weights the importance of the similarity loss in (24) with respect to the prediction loss; $\gamma$, which balances the weights between the pairwise term and the unary term [see (14)]; and the number of mean-field iterations, equivalently, the number of mean-field layers, $T$. We carry out cross validation on a separate split of the MovieLens dataset ($75\%$ training, $5\%$ validation and $20\%$ testing). This split is randomly generated and is different from that used to compare the proposed model against the other models.

V-B1 Weight of the Similarity Loss Term

To determine the best value for $\beta$, we first fix $T$ to $30$, with which the mean-field inference is likely to converge [57], and set $\gamma$ empirically to $0.01$. As both the computed and the ground-truth entry similarities lie in the range $[0,1]$, the similarity loss is normally much smaller than the prediction loss. As a result, we experiment with small and large values of $\beta$ in $[0.0,3.0]$. The results of this experiment are shown in Table. II.

It can be seen that $\beta=1.5$ gives the best performance. Furthermore, when $\beta>0$, the RMSE errors drop significantly compared to when $\beta=0$. Recall that when $\beta=0$, the similarity loss has no effect during training. This proves the benefit of supervising the similarity learning using the proposed method.

V-B2 The value of $\gamma$

We fix $\beta$ to $1.5$, which is found in the previous experiment, $T$ to $30$, and run the proposed model using different values of $\gamma$ from $0.01$ to $0.5$. The results of this experiment are shown in Table III.

As can be seen, $\gamma=0.05$ yields the best performance. The predictions become less accurate when $\gamma$ becomes large (beyond $0.25$), possibly because the pairwise terms start to dominate the unary terms.

V-B3 Number of Mean-field Iterations

Fixing $\beta=1.5$ and $\gamma=0.05$, we then run the DCMC model with different numbers of mean-field iterations. The results of this experiment are summarized in Table IV.

It can be observed that the RMSE improves as $T$ increases. Even though we still observe improvements when $T$ is larger than $5$, the differences are very small. Therefore, we select $T=5$ as it provides the best trade-off between accuracy and computational complexity.

V-C Comparison Against State-of-the-art Models

After finding the most effective hyperparameter settings, we carry out experiments to compare the proposed model with reference models on the five real-world datasets. We select state-of-the-art deep-learning-based matrix completion models as references, including non-geometric models: the item-based and user-based autoencoders (I-Autorec and U-Autorec) [18], the deep User-based autoencoder (Deep U-Autorec) [20], the deep matrix factorization model (NMC) [22], the manifold-learning-based-regularized autoencoders (m-I-Autorec and m-U-Autorec) [21], the CF-NADE model [58]; and geometric models: the sRGCNN model [33] and the GCMC model [34]. For these reference models, we use the source codes released by their authors. For the sRGCNN and GCMC models, the graphs are constructed from the observed ratings. We run each model five times and report the average RMSE and MAE values, together with their standard deviations.

Table V and Table VI present the results for different models on the MovieLens dataset, and on the Flixster, Douban and YahooMusic datasets, respectively. On the MovieLens dataset, the proposed model outperforms all other models in both scores, followed by the m-I-Autorec [21] and the CF-NADE [58] models. On the Flixster dataset, the I-Autorec model yields the best performance, while our DCMC model is ranked second. On both the Douban and YahooMusic datasets, our model consistently outperforms the reference models. We do not include the results of the CF-NADE model on the YahooMusic dataset, as it requires an excessive amount of memory, proportional to the number of rating levels ($100$ in this case).

We further compare the performance of the models on the Epinions dataset [56], which is of much higher scale than the other datasets used in the experiments. Another challenge is that in this dataset, the given observations are highly scarse with respect to the large matrix dimensions. Table VII presents the results of different models on this dataset. We do not include the sRGCNN [33] and the CF-NADE [58] models as they do not scale well to this dataset.

It can be seen that our model outperforms the reference models on this dataset, in terms of both the RMSE and MAE scores, whereas the U-Autorec model has the second best performance.

As mentioned earlier, the design of the base prediction network in the DCMC model follows that of the NMC model [22]. Even though the NMC model performs relatively well on the MovieLens dataset, its performance deteriorates on the Flixster, YahooMusic and Epinions datasets, where the numbers of observed entries are highly limited. By effectively learning and leveraging the relationships among entries, the DCMC model significantly improves the accuracy over the NMC model on these datasets. It is evident that over the benchmark datasets, the DCMC model consistently reports low prediction errors and achieves the best overall performance among all the models. The performance gains brought by the DCMC model are more profound as the data becomes highly scarce (e.g., on the YahooMusic and Epinions datasets).

V-D Effects of Training the Base Prediction Network with the Mean-field Inference

In Section IV-D, we argued the advantage of the proposed model over a two-stage method. To verify this argument, we perform an experiment comparing the results when using different training/testing variants. The first variant involves training and testing without the mean-field inference. This is equivalent to using only the base prediction network in both training and testing. The second variant involves training without and testing with the mean-field inference. This is equivalent to a two-stage approach, running the base prediction network to compute the CRF potentials and then run the mean-field algorithm. The third variant involves training with and testing without the mean-field inference. This variant allows us to see the effects of training the base prediction network with feedback from the mean-field inference. The last variant is our final DCMC model, which applies training and testing with inference in an end-to-end manner. The same set of hyperparameters is used for all the variants. We use the learned similarities for the variants with the mean-field inference in the testing phase.

The results of this experiment are summarized in Table VIII. It is clear that using the mean-field inference in testing improves the performance independent of whether the model is trained with or without mean-field inference. This shows the benefit of using the mean-field inference with the learned similarities, to gather the information from the predicting entries when making prediction for a missing entry. Training the base prediction network with feedback from the mean-field inference and then testing it without mean-field inference degrades the performance. However, training and testing with the mean-field inference (the DCMC model) yields the best performance. This shows the benefits of the proposed end-to-end training over the two-stage approach.

V-E Quality of the Learned Similarities

The DCMC model learns the similarities between users and items, and in turn computes the similarities between entries. In this sub-section, we evaluate the capacity of the model to learn the entry similarities, since the quality of these learned similarities has a strong impact on the prediction accuracy.

We follow an indirect evaluation where we compare the prediction error of the benchmark $k$-NN method, specifically, its user-based and item-based variants, when using the learned user and item similarities—obtained by running our approach on the datasets—against that when using widely-used similarity metrics. We select four similarity metrics for this comparison, namely, the cosine similarity (cosine), the mean square difference (msd), the Pearson correlation coefficient (pearson) [6], and the shrunk Pearson correlation coefficient (pearson_shrunk) [5]. We employ the implementations of the $k$-NN method and the pre-defined similarity metrics in the Surprise recommendation system library111https://surprise.readthedocs.io/ (in this library, the $k$-NN method is called “KNNBasic”).

Fig. 5 shows the RMSE values obtained when using the user- and item-based $k$-NN methods with the five approaches to compute the user and item similarities, with $k$ varying in $[10,300]$. Evidently, the proposed learned similarities lead to the best performance independently of the $k$ value. On the MovieLens dataset, the benefit of using the learned similarities is less evident than on the YahooMusic dataset. The reason is that only less than $0.06\%$ of the entries on the YahooMusic dataset are observed. As such, all the pre-defined metrics become less reliable and the $k$-NN method suffers when using these metrics to calculate user and item similarities. We observe the same patterns when performing this experiment on the Flixster and Douban datasets. This shows the benefit of using the proposed model to learn the user and item similarities, especially from a very limited number of observations.

VI Conclusion

In this paper, we formulated matrix completion as a MAP inference problem in a CRF. The inference problem was solved using the mean-field algorithm. By unfolding the mean-field algorithm into specially-designed neural network layers, we constructed a deep model that simultaneously computes the CRF potentials, learns the correlations among the nodes in the CRF and performs the mean-field inference in each forward pass. The model can be trained in an end-to-end manner, using a method to supervise the learning of the similarities between entries. Experimental studies using various real-world datasets showed that the proposed model consistently yields better performance than various state-of-the-art models, especially on datasets with very limited number of observations, and justified the benefits of each of the proposed components.