Secure Federated Matrix Factorization

Di Chai; Leye Wang; Kai Chen; Qiang Yang

arXiv:1906.05108·cs.CR·April 12, 2022

Secure Federated Matrix Factorization

Di Chai, Leye Wang, Kai Chen, Qiang Yang

PDF

TL;DR

This paper introduces FedMF, a federated matrix factorization framework that enhances user privacy through homomorphic encryption, enabling secure model training without raw data sharing, and demonstrates its feasibility with real data.

Contribution

We propose FedMF, a novel federated matrix factorization method that incorporates homomorphic encryption to protect user data during distributed learning.

Findings

01

FedMF is feasible with real-world datasets.

02

Gradient information can leak raw user data.

03

Homomorphic encryption enhances privacy in federated matrix factorization.

Abstract

To protect user privacy and meet law regulations, federated (machine) learning is obtaining vast interests in recent years. The key principle of federated learning is training a machine learning model without needing to know each user's personal raw private data. In this paper, we propose a secure matrix factorization framework under the federated learning setting, called FedMF. First, we design a user-level distributed matrix factorization framework where the model can be learned when each user only uploads the gradient information (instead of the raw preference data) to the server. While gradient information seems secure, we prove that it could still leak users' raw data. To this end, we enhance the distributed matrix factorization framework with homomorphic encryption. We implement the prototype of FedMF and test it with a real movie rating dataset. Results verify the feasibility of…

Figures3

Click any figure to enlarge with its caption.

Tables1

Table 1. Table 1: Time consumption of each iteration (seconds).

#Item	#Rating	PartText	FullText
40	8307	34.39	90.94
50	9807	44.05	113.34
60	11214	46.34	141.52
80	13817	52.91	182.27
160	22282	92.81	374.85
320	34172	140.51	725.72
640	49706	178.24	1479.40
1280	67558	264.10	2919.91
2560	83616	334.79	5786.01

Equations55

E (m_{1}) ⋆ E (m_{2}) = E (m_{1} ⋆ m_{2}), \forall m_{1}, m_{2} \in M

E (m_{1}) ⋆ E (m_{2}) = E (m_{1} ⋆ m_{2}), \forall m_{1}, m_{2} \in M

U, V min \frac{1}{M} (r_{i, j} - ⟨ u_{i}, v_{j} ⟩)^{2} + λ ∣∣ U ∣ ∣_{2}^{2} + μ ∣∣ V ∣ ∣_{2}^{2}

U, V min \frac{1}{M} (r_{i, j} - ⟨ u_{i}, v_{j} ⟩)^{2} + λ ∣∣ U ∣ ∣_{2}^{2} + μ ∣∣ V ∣ ∣_{2}^{2}

u_{i}^{t} = u_{i}^{t - 1} - γ ▽_{u_{i}} F (U^{t - 1}, V^{t - 1})

u_{i}^{t} = u_{i}^{t - 1} - γ ▽_{u_{i}} F (U^{t - 1}, V^{t - 1})

v_{i}^{t} = v_{i}^{t - 1} - γ ▽_{v_{i}} F (U^{t - 1}, V^{t - 1})

v_{i}^{t} = v_{i}^{t - 1} - γ ▽_{v_{i}} F (U^{t - 1}, V^{t - 1})

▽_{u_{i}} F (U, V) = - 2 j : (i, j) \sum v_{j} (r_{ij} - ⟨ u_{i}, v_{j} ⟩) + 2 λ u_{i}

▽_{u_{i}} F (U, V) = - 2 j : (i, j) \sum v_{j} (r_{ij} - ⟨ u_{i}, v_{j} ⟩) + 2 λ u_{i}

▽_{v_{j}} F (U, V) = - 2 i : (i, j) \sum u_{i} (r_{ij} - ⟨ u_{i}, v_{j} ⟩) + 2 λ v_{j}

▽_{v_{j}} F (U, V) = - 2 i : (i, j) \sum u_{i} (r_{ij} - ⟨ u_{i}, v_{j} ⟩) + 2 λ v_{j}

u_{i}^{t} (r_{ij} - ⟨ u_{i}^{t}, v_{j}^{t} ⟩) = G_{j}^{t}, j \in M_{i}

u_{i}^{t} (r_{ij} - ⟨ u_{i}^{t}, v_{j}^{t} ⟩) = G_{j}^{t}, j \in M_{i}

u_{i}^{t + 1} (r_{ij} - ⟨ u_{i}^{t + 1}, v_{j}^{t + 1} ⟩) = G_{j}^{t + 1}, j \in M_{i}

u_{i}^{t + 1} (r_{ij} - ⟨ u_{i}^{t + 1}, v_{j}^{t + 1} ⟩) = G_{j}^{t + 1}, j \in M_{i}

u_{i}^{t} - u_{i}^{t + 1} = - 2 j \in M_{i} \sum v_{j}^{t} (r_{ij} - ⟨ u_{i}^{t}, v_{j}^{t} ⟩)

u_{i}^{t} - u_{i}^{t + 1} = - 2 j \in M_{i} \sum v_{j}^{t} (r_{ij} - ⟨ u_{i}^{t}, v_{j}^{t} ⟩)

⎩ ⎨ ⎧ u_{i 1}^{t} (r_{ij} - \sum_{m = 1}^{D} u_{im}^{t} v_{j m}^{t}) = G_{j 1}^{t} ⋮ u_{ik}^{t} (r_{ij} - \sum_{m = 1}^{D} u_{im}^{t} v_{j m}^{t}) = G_{j k}^{t} ⋮ u_{i D}^{t} (r_{ij} - \sum_{m = 1}^{D} u_{im}^{t} v_{j m}^{t}) = G_{j D}^{t}

⎩ ⎨ ⎧ u_{i 1}^{t} (r_{ij} - \sum_{m = 1}^{D} u_{im}^{t} v_{j m}^{t}) = G_{j 1}^{t} ⋮ u_{ik}^{t} (r_{ij} - \sum_{m = 1}^{D} u_{im}^{t} v_{j m}^{t}) = G_{j k}^{t} ⋮ u_{i D}^{t} (r_{ij} - \sum_{m = 1}^{D} u_{im}^{t} v_{j m}^{t}) = G_{j D}^{t}

⎩ ⎨ ⎧ u_{i 1}^{t + 1} (r_{ij} - \sum_{m = 1}^{D} u_{im}^{t + 1} v_{j m}^{t + 1}) = G_{j 1}^{t + 1} ⋮ u_{ik}^{t + 1} (r_{ij} - \sum_{m = 1}^{D} u_{im}^{t + 1} v_{j m}^{t + 1}) = G_{j k}^{t + 1} ⋮ u_{i D}^{t + 1} (r_{ij} - \sum_{m = 1}^{D} u_{im}^{t + 1} v_{j m}^{t + 1}) = G_{j D}^{t + 1}

⎩ ⎨ ⎧ u_{i 1}^{t + 1} (r_{ij} - \sum_{m = 1}^{D} u_{im}^{t + 1} v_{j m}^{t + 1}) = G_{j 1}^{t + 1} ⋮ u_{ik}^{t + 1} (r_{ij} - \sum_{m = 1}^{D} u_{im}^{t + 1} v_{j m}^{t + 1}) = G_{j k}^{t + 1} ⋮ u_{i D}^{t + 1} (r_{ij} - \sum_{m = 1}^{D} u_{im}^{t + 1} v_{j m}^{t + 1}) = G_{j D}^{t + 1}

⎩ ⎨ ⎧ u_{i 1}^{t} - u_{i 1}^{t + 1} = - 2 \sum_{n = 1}^{N} v_{n 1}^{t} (r_{in} - \sum_{m = 1}^{D} u_{im}^{t} v_{nm}^{t}) ⋮ u_{ik}^{t} - u_{ik}^{t + 1} = - 2 \sum_{n = 1}^{N} v_{nk}^{t} (r_{in} - \sum_{m = 1}^{D} u_{im}^{t} v_{nm}^{t}) ⋮ u_{i D}^{t} - u_{i D}^{t + 1} = - 2 \sum_{n = 1}^{N} v_{n D}^{t} (r_{in} - \sum_{m = 1}^{D} u_{im}^{t} v_{nm}^{t})

⎩ ⎨ ⎧ u_{i 1}^{t} - u_{i 1}^{t + 1} = - 2 \sum_{n = 1}^{N} v_{n 1}^{t} (r_{in} - \sum_{m = 1}^{D} u_{im}^{t} v_{nm}^{t}) ⋮ u_{ik}^{t} - u_{ik}^{t + 1} = - 2 \sum_{n = 1}^{N} v_{nk}^{t} (r_{in} - \sum_{m = 1}^{D} u_{im}^{t} v_{nm}^{t}) ⋮ u_{i D}^{t} - u_{i D}^{t + 1} = - 2 \sum_{n = 1}^{N} v_{n D}^{t} (r_{in} - \sum_{m = 1}^{D} u_{im}^{t} v_{nm}^{t})

\frac{u _{ik}^{t}}{u _{i (k + 1)}^{t}} = \frac{G _{ik}^{t}}{G _{i (k + 1)}^{t}}

\frac{u _{ik}^{t}}{u _{i (k + 1)}^{t}} = \frac{G _{ik}^{t}}{G _{i (k + 1)}^{t}}

r_{ij} - m = 1 \sum D u_{im}^{t} v_{j m}^{t} = \frac{G _{j k}^{t}}{u _{ik}^{t}}

r_{ij} - m = 1 \sum D u_{im}^{t} v_{j m}^{t} = \frac{G _{j k}^{t}}{u _{ik}^{t}}

u_{ik}^{t} - u_{ik}^{t + 1} = - 2 \frac{1}{u _{ik}^{t}} n = 1 \sum N v_{nk}^{t} G_{nk}^{t}

u_{ik}^{t} - u_{ik}^{t + 1} = - 2 \frac{1}{u _{ik}^{t}} n = 1 \sum N v_{nk}^{t} G_{nk}^{t}

u_{ik}^{t + 1} = u_{ik}^{t} + 2 \frac{1}{u _{ik}^{t}} n = 1 \sum N v_{nk}^{t} G_{nk}^{t}

u_{ik}^{t + 1} = u_{ik}^{t} + 2 \frac{1}{u _{ik}^{t}} n = 1 \sum N v_{nk}^{t} G_{nk}^{t}

\frac{G _{j k}^{t}}{u _{ik}^{t}} + m = 1 \sum D u_{im}^{t} v_{j m}^{t} = \frac{G _{j k}^{t + 1}}{u _{ik}^{t + 1}} + m = 1 \sum D u_{im}^{t + 1} v_{j m}^{t + 1}

\frac{G _{j k}^{t}}{u _{ik}^{t}} + m = 1 \sum D u_{im}^{t} v_{j m}^{t} = \frac{G _{j k}^{t + 1}}{u _{ik}^{t + 1}} + m = 1 \sum D u_{im}^{t + 1} v_{j m}^{t + 1}

\frac{G _{j k}^{t}}{u _{ik}^{t}} + m = 1 \sum D u_{im}^{t} v_{j m}^{t} = \frac{G _{j k}^{t + 1}}{u _{ik}^{t} + 2 \frac{1}{u _{ik}^{t}} \sum _{n = 1}^{N} v _{nk}^{t} G _{nk}^{t}} +

\frac{G _{j k}^{t}}{u _{ik}^{t}} + m = 1 \sum D u_{im}^{t} v_{j m}^{t} = \frac{G _{j k}^{t + 1}}{u _{ik}^{t} + 2 \frac{1}{u _{ik}^{t}} \sum _{n = 1}^{N} v _{nk}^{t} G _{nk}^{t}} +

m = 1 \sum D (u_{im}^{t} + 2 \frac{1}{u _{im}^{t}} n = 1 \sum N v_{nm}^{t} G_{nm}^{t}) v_{j m}^{t + 1}

\frac{G _{j k}^{t}}{u _{ik}^{t}} - \frac{G _{j k}^{t + 1}}{u _{ik}^{t} + 2 \frac{1}{u _{ik}^{t}} \sum _{n = 1}^{N} v _{nk}^{t} G _{nk}^{t}}

\frac{G _{j k}^{t}}{u _{ik}^{t}} - \frac{G _{j k}^{t + 1}}{u _{ik}^{t} + 2 \frac{1}{u _{ik}^{t}} \sum _{n = 1}^{N} v _{nk}^{t} G _{nk}^{t}}

= m = 1 \sum D [(u_{im}^{t} + 2 \frac{1}{u _{im}^{t}} n = 1 \sum N (v_{nm}^{t} G_{nm}^{t})) v_{j m}^{t + 1} - u_{im}^{t} v_{j m}^{t}]

\frac{G _{j k}^{t}}{u _{ik}^{t}} - \frac{G _{j k}^{t + 1}}{u _{ik}^{t} + \frac{α _{k}}{u _{ik}^{t}}}

\frac{G _{j k}^{t}}{u _{ik}^{t}} - \frac{G _{j k}^{t + 1}}{u _{ik}^{t} + \frac{α _{k}}{u _{ik}^{t}}}

= m = 1 \sum D [(v_{j m}^{t + 1} - v_{j m}^{t}) u_{im}^{t} + \frac{α _{m} v _{j m}^{t + 1}}{u _{im}^{t}}]

u_{im}^{t} = \frac{G _{j m}^{t}}{G _{j k}^{t}} u_{ik}^{t}

u_{im}^{t} = \frac{G _{j m}^{t}}{G _{j k}^{t}} u_{ik}^{t}

\frac{G _{j k}^{t}}{u _{ik}^{t}} - \frac{G _{j k}^{t + 1}}{u _{ik}^{t} + \frac{α _{k}}{u _{ik}^{t}}}

\frac{G _{j k}^{t}}{u _{ik}^{t}} - \frac{G _{j k}^{t + 1}}{u _{ik}^{t} + \frac{α _{k}}{u _{ik}^{t}}}

= m = 1 \sum D [(v_{j m}^{t + 1} - v_{j m}^{t}) \frac{G _{j m}^{t}}{G _{j k}^{t}} u_{ik}^{t} + \frac{α _{m} v _{j m}^{t + 1}}{\frac{G _{j m}^{t}}{G _{j k}^{t}} u _{ik}^{t}}]

= \frac{u _{ik}^{t}}{G _{j k}^{t}} m = 1 \sum D [(v_{j m}^{t + 1} - v_{j m}^{t}) G_{j m}^{t}] + \frac{G _{j k}^{t}}{u _{ik}^{t}} m = 1 \sum D [\frac{α _{m} v _{j m}^{t + 1}}{G _{j m}^{t}}]

⎩ ⎨ ⎧ β_{j} = \sum_{m = 1}^{D} [(v_{j m}^{t + 1} - v_{j m}^{t}) G_{j m}^{t}] γ_{j} = \sum_{m = 1}^{D} [\frac{α _{m} v _{j m}^{t + 1}}{G _{j m}^{t}}]

⎩ ⎨ ⎧ β_{j} = \sum_{m = 1}^{D} [(v_{j m}^{t + 1} - v_{j m}^{t}) G_{j m}^{t}] γ_{j} = \sum_{m = 1}^{D} [\frac{α _{m} v _{j m}^{t + 1}}{G _{j m}^{t}}]

\frac{G _{j k}^{t}}{u _{ik}^{t}} - \frac{G _{j k}^{t + 1}}{u _{ik}^{t} + \frac{α _{k}}{u _{ik}^{t}}} = \frac{u _{ik}^{t}}{G _{j k}^{t}} β_{j} + \frac{G _{j k}^{t}}{u _{ik}^{t}} γ_{j}

\frac{G _{j k}^{t}}{u _{ik}^{t}} - \frac{G _{j k}^{t + 1}}{u _{ik}^{t} + \frac{α _{k}}{u _{ik}^{t}}} = \frac{u _{ik}^{t}}{G _{j k}^{t}} β_{j} + \frac{G _{j k}^{t}}{u _{ik}^{t}} γ_{j}

r_{ij} = \frac{G _{j k}^{t}}{u _{ik}^{t}} + m = 1 \sum D u_{im}^{t} v_{j m}^{t}

r_{ij} = \frac{G _{j k}^{t}}{u _{ik}^{t}} + m = 1 \sum D u_{im}^{t} v_{j m}^{t}

Peer Reviews

No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.

Videos

No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.

Full text

Secure Federated Matrix Factorization

Di Chai1,2,∗

Leye Wang3,∗

Kai Chen1

Qiang Yang1,4 1Hong Kong University of Science and Technology, China; 2Clustar, China

3Peking University, China; 4WeBank, China

[email protected], [email protected], [email protected], [email protected]

∗*equal contribution, ranked alphabetically *

Abstract

To protect user privacy and meet law regulations, federated (machine) learning is obtaining vast interests in recent years. The key principle of federated learning is training a machine learning model without needing to know each user’s personal raw private data. In this paper, we propose a secure matrix factorization framework under the federated learning setting, called FedMF. First, we design a user-level distributed matrix factorization framework where the model can be learned when each user only uploads the gradient information (instead of the raw preference data) to the server. While gradient information seems secure, we prove that it could still leak users’ raw data. To this end, we enhance the distributed matrix factorization framework with homomorphic encryption. We implement the prototype of FedMF and test it with a real movie rating dataset. Results verify the feasibility of FedMF. We also discuss the challenges for applying FedMF in practice for future research.

1 Introduction

With the prevalence of government regulations and laws on privacy protection in the big data era (e.g., General Data Protection Regulation111https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX:32016R0679R(02)), privacy-preserving machine learning has obtained rapidly growing interests in both academia and industry. Among various techniques to achieve privacy-preserving machine learning, federated (machine) learning (FL) recently receives high attention. The original idea of FL was proposed by Google Konečnỳ et al. (2016), which targets at learning a centered model based on the personal information distributed at each user’s mobile phone. More importantly, during model training, no user’s raw personal information is transferred to the central server, thus ensuring the privacy protection. Also, the learned privacy-preserving model can be proved to hold almost similar predictive power compared to the traditional model learned on users’ raw data. This highlights the practicality of FL as little predictive accuracy is sacrificed, especially compared to other accuracy-lossy privacy preserving mechanisms such as differential privacy Dwork (2011).

Starting from the original Google paper, many researchers have been devoted into this promising and critical area. Recently, a nice survey paper on FL has been published Yang et al. (2019). While many research works have been done on FL, a popular machine learning technique, matrix factorization (MF) Koren et al. (2009), is still under-investigated in FL. Since MF is one of the prominent techniques widely employed in various applications such as item recommendation Koren et al. (2009) and environment monitoring Wang et al. (2016), we highly believe that studying MF under FL is urgently required. This work is one of the pioneering research efforts toward this direction.

Taking recommendation systems as an example, two types of users’ private information are leaked in traditional MF Koren et al. (2009): (i) users’ raw preference data, and (ii) users’ learned latent feature vectors. As revealed by previous studies, either raw preference data or latent features can leak users’ sensitive attributes, e.g., age, relationship status, political views, and sex orientations Yang et al. (2016); Kosinski et al. (2013). This highlights the importance of protecting users’ private information during MF. Prior studies have studied privacy-preserving MF in two main types:

(1) Obfuscation-based methods obfuscate users’ preference raw data before releasing it to the central server so as to ensure certain level of privacy protection (e.g., differential privacy) Berlioz et al. (2015). The pitfall is that obfuscation inevitably leads to the loss of predictive power of the learned latent feature vectors. Hence, these methods usually need to make a trade-off between the privacy protection and the model performance.

(2) Encryption-based methods use advanced encryption schemes such as homomorphic encryption for implementing privacy-preserving MF Kim et al. (2016). While they usually do not need to sacrifice predictive power for privacy protection, they commonly require a third-party crypto-service provider. This makes the system implementation complicated as such a provider is not easy to find in practice.222Some studies suggest that governments can perform this role, but this is still not the case in reality now Kim et al. (2016). Moreover, if the crypto-service provider collude with the recommendation server, then no user privacy protection can be preserved Kim et al. (2016); Nikolaenko et al. (2013).

This research proposes a novel FL-based MF framework, called FedMF (Federated Matrix Factorization). FedMF employs distributed machine learning and homomorphic encryption schemes. In brief, FedMF lets each user compute the gradients of his/her own rating information locally and then upload the gradients (instead of the raw data) to the server for training. To further enhance security, each user can encrypt the gradients with homomorphic encryption. With FedMF, two shortcomings of the traditional obfuscation- or encryption-based methods can be addressed: (i) no predictive accuracy is lost as we do not obfuscate data; (ii) no third-party crypto-service provider is required as each user’s device can handle the secure gradient computing task.

In summary, we have the following contributions:

(1) To the best of our knowledge, we design the first FL-based secure MF framework, called FedMF, which can overcome the shortcomings of traditional obfuscation- or encryption-based mechanisms as aforementioned.

(2) To implement FedMF, first, we design a user-level distributed matrix factorization framework where the model can be learned when each user only uploads the gradient information (instead of the raw preference data) to the server. Though gradient information seems secure, we prove that it could still leak users’ raw data to some extent. Then, we enhance the distributed matrix factorization framework with homomorphic encryption to increase the security.

(3) We implement a prototype of FedMF (the code will be published in https://github.com/Di-Chai/FedMF). We test the prototype on a real movie rating dataset. Results verify the feasibility of FedMF.

It is worth noting that, similar to FedMF, Ammad-ud-din et al. (2019) tried to develop a federated collaborative filtering system. However, Ammad-ud-din et al. (2019) directly let users upload their gradient information to the server. As we will prove in this paper, gradients can still reveal users’ original preference data. Hence, a more secure system like FedMF is required to rigorously protect users’ privacy.

2 Preliminaries

In this section, we briefly introduce two techniques closely related to our research, horizontal federated learning and additively homomorphic encryption.

2.1 Horizontal Federated Learning

Federated learning is a method that enables a group of data owners to train a machine learning model on their joint data and nobody can learn the data of the participants. Federated learning can be categorized based on the distribution characteristics of the data Yang et al. (2019). One category of federated learning is horizontal federated learning. Horizontal federated learning is introduced in the scenarios when the data from different contributors share the same feature space but vary in samples. In our secure matrix factorization recommendation system, the rating information distributed on each user’s device has exactly the same feature space, while different users are exactly different samples. So our matrix factorization in federated learning can be categorized as horizontal federated learning. Following the typical assumption of horizontal federated learning Yang et al. (2019), we assume all the users are honest and the system is designed to be secure against an honest-but-curious server.

2.2 Additively Homomorphic Encryption

Homomorphic encryption (HE) is often used in federated learning to protect user’s privacy by providing encrypted parameter exchange. HE is a special kind of encryption scheme that allows any third party to operate on the encrypted data without decrypting it in advance. An encryption scheme is called homomorphic over an operation ’ $\star$ ’ if the following equation holds:

[TABLE]

where $E$ is an encryption algorithm and $M$ is the set of all possible messages Acar et al. (2018).

Addictively homomorphic encryption is homomorphic over addition. Typically, it consists of the following functions:

•

$KeyGen\rightarrow(pk,sk)$ : key generation process, where $pk$ is the public key and $sk$ is the secret key.

•

$Enc(m,pk)\rightarrow c$ : encryption process, where $m$ is the message to be encrypted and c is the ciphertext.

•

$Dec(c,sk)\rightarrow m$ : decryption process

•

$Add(c_{1},c_{2})\rightarrow c_{a}(i.e.\ Enc(c_{1}+c_{2}))$ : add operation on ciphertext, $c_{a}$ is the ciphertext of plaintexts’ addition.

•

$DecAdd(c_{a},sk)\rightarrow m_{a}$ : decrypt $c_{a}$ , getting the addition of plaintexts.

3 User-level Distributed Matrix Factorization

We firstly introduce the matrix factorization optimization method used in our paper, stochastic gradient descent Koren et al. (2009). Based on it, we design a user-level distributed matrix factorization framework.

3.1 Stochastic Gradient Descent

Suppose we have $n$ users, $m$ items and each user rated a subset of $m$ items. For $[n]:=\{1,2...,n\}$ as the set of users and $[m]:=\{1,2,...,m\}$ as the set of items, we denote $\mathcal{M}\in[n]\times[m]$ as user-item rating pairs which a rating has been generated, $M=|\mathcal{M}|$ as the total number of ratings and $r_{i,j}$ represents the rating generated by user $i$ for item $j$ .

Given the rating information ${r_{ij}:(i,j)\in\mathcal{M}}$ , the recommendation systems are expected to predict the rating values of all the items for all the users. Matrix factorization formulates this problem as fitting a bi-linear model on the existing ratings. In particular, user profile matrix $U\in\mathbf{R}^{n\times d}$ and item profile matrix $V\in\mathbf{R}^{m\times d}$ are computed, the resulting profile matrices are used to predict user $i$ ’s rating on item $j$ , which is $\langle u_{i},v_{j}\rangle$ . The computing process of $U$ and $V$ can be done by solving the following regularized least squares minimization:

[TABLE]

where $\lambda$ and $\mu$ are small positive values to rescale the penalizer. Stochastic gradient descent iteratively updates $U$ and $V$ with the following equations Koren et al. (2009):

[TABLE]

where

[TABLE]

The number of iterations relies on the stopping criteria. A typical criteria is to set a small threshold $\varepsilon$ , such that the training stops when the gradient $\bigtriangledown_{u_{i}}F$ and $\bigtriangledown_{v_{j}}F$ (or one of them) are smaller than $\varepsilon$ .

3.2 Distributed Matrix Factorization

In the distributed matrix factorization scenario, users hold their rating information locally and the model is trained on their joint data. To achieve this goal, we leverage a distributed matrix factorization method, which decomposes the iterative updating process into two parts that are performed on the user side and the server side, respectively. In particular, equation (3) is executed on user $i$ ’s device, namely user update, and equation (4) is performed on the server, called server update. This decomposition prevents the server from directly knowing users’ raw preference data or learned profiles.

Algorithm 1 shows our user-level distributed matrix factorization method. The server keeps providing the latest item profile matrix $V$ for all the users to download. Having the latest downloaded $V$ and his/her own rating information, each user $i$ performs local updates and computes $Gradient_{i}$ , which will be sent back to the server to update item profiles.

4 Gradients Leak Information

Algorithm 1 shows a framework that allows the server to build a matrix factorization recommendation system on a distributed dataset, i.e. users keep rating information locally. In this framework, users iteratively send $Gradient$ information to server in plain text. Next we are going to prove that such a distributed matrix factorization system cannot protect users’ rating information against the server, i.e. server can deduce users’ rating data using the $Gradient$ .

For user-vector $u_{i}$ , suppose the user rated item-set is $M_{i}$ . We will have the following equation at time $t$

[TABLE]

where $G$ is the gradient to be uploaded from the user $i$ to the server for updating item-profiles. Similarly, at time $t+1$ ,

[TABLE]

The correlation between $U_{t}$ and $U_{t+1}$ is :

[TABLE]

Take a close look at the element-wise calculation of equation (7)–(9), where we denote the latent user and item vectors ( $u_{i}$ and $v_{j}$ ) are $D$ dimension:

[TABLE]

Now we turn to analyze the $k$ -th entry of $u_{i}$ , $u_{ik}$ . From equation (10), we have :

[TABLE]

Plug equation (13) into (11), we will have

[TABLE]

Thus we can represent $u^{t+1}_{ik}$ using $u^{t}_{ik}$ as:

[TABLE]

From equation (10) and (11) we have:

[TABLE]

Plug equation (16) into (17):

[TABLE]

which is,

[TABLE]

Let $\alpha_{k}=2\sum^{N-1}_{n=0}v^{t}_{nk}G_{nk}^{t}$ ,

[TABLE]

From equation (13), we can have:

[TABLE]

Plug equation (21) into (4):

[TABLE]

Denote $\beta_{j}$ and $\gamma_{j}$ as follow:

[TABLE]

We will have:

[TABLE]

Since we know there must be one real scalar of $u^{t}_{ik}$ that satisfies equation (24). We can use some iterative methods to compute a numeric solution of (24), e.g., Newton’s method.

After getting $u^{t}_{i}$ , we can use equation (10) to compute $r_{i}$ , which can be written as:

[TABLE]

In summary, knowing the gradients of a user uploaded in two continuous steps, we can infer this user’s rating information. Thus, we propose a secure matrix factorization framework based on homomorphic encryption, which will be elaborated in the next section.

5 FedMF: Federated Matrix Factorization

To overcome this information leakage problem, we propose to encode the gradients such that server cannot inverse the encoding process. Then, the encoded data leaks no information. Meanwhile, the server should still be able to perform updates using the encoded gradients. One way to achieve such a goal is using homomorphic encryption.

Figure 1 shows a framework of our method, called FedMF (Federated Matrix Factorization). Two types of participants are involved in this framework, the server and the users. As previously illustrated in Sec. 2.1, we assume that the server is honest-but-curious, the users are honest, and the privacy of the users is protected against the server.

Key Generation: As the typical functions involved in homomorphic encryption (Sec. 2.2), we first generate the public key and secret key. The key generation process is carried out on one of the users. The public key is known to all the participants including the server. And the secret key is only shared between users and needs to be protected against the server. After the keys are generated, different TLS/SSL secure channels will be established for sending the public key and secret key to the corresponding participants.

Parameter Initialization: Before starting the matrix factorization process, some parameters need to be initialized. The item profile matrix is initialized at the server side while the user profile matrix is initialized by each user locally.

Matrix Factorization: Major steps include,

The server encrypts item profile $V$ using public key, getting the ciphertext $C_{V}$ . From now on, the latest $C_{V}$ is prepared for all users’ download. 2. 2.

Each user downloads the latest $C_{V}$ from the server, and decrypts it using secret key, getting the plaintext of $V$ . $V$ is used to perform local update and compute the gradient $G$ . Then $G$ is encrypted using public key, getting ciphertext $C_{V}$ . Then a TLS/SSL secure channel is built, $C_{V}$ is sent back to the server via this secure channel. 3. 3.

After receiving a user’s encrypted gradient, the server updates the item profile using this ciphertext : $C_{V}^{t+1}=C_{V}^{t}-C_{G}$ . Afterwards, the latest $C_{V}$ is prepared for users’ downloading. 4. 4.

Step 2 and 3 are iteratively executed until convergence.

Security against Server: As shown in Fig. 1, only ciphertext is sent to the server in FedMF. So no bit of information will be leaked to the server as long as our homomorphic encryption system ensures ciphertext indistinguishability against chosen plaintext attacks Goldreich (2009).

No Accuracy Decline: We also claim that FedMF is accuracy equivalent to the user-level distributed matrix factorization. This is because the parameter updating process is same as the distributed matrix factorization (Sec. 3) if the homomorphic encryption part is removed.

6 Prototype and Evaluation

In this section, we choose Paillier encryption method Paillier (1999) to instantiate a prototype of our system and use a real movie rating dataset to evaluate the prototype.

6.1 Prototype Implementation

We use Paillier encryption Paillier (1999) to build a prototype of FedMF. Paillier encryption is a probabilistic encryption schema based on composite residuosity problem Jager (2012). Given publick key and encrypted plaintext, paillier encryption has the following homomorphic property operations:

op1. $E(m_{1})\cdot E(m_{2})\ (mode\ n^{2})=E(m_{1}+m_{2}\ (mode\ n))$

op2. $E(m_{1})\cdot g^{m_{2}}\ (mode\ n^{2})=E(m_{1}+m_{2}\ (mode\ n))$

op3. $E(m_{1})^{m_{2}}\ (mode\ n^{2})=E(m_{1}m_{2}\ (mode\ n))$

Typically, paillier encryption requires the plaintext to be positive integer. But in our system, the data are all in the form of floating point numbers and some of them might be negative. Thus we need to extend the encryption method to support our system.

Float. In brief, a base exponent was multiplied to the decimal, the integer part of the multiplication result $I$ and the base exponent $e$ was treated as the integer representation of the floating point number, i.e. $(I,e)$ . In the encryption process, only $I$ is encrypted, the ciphertext will be $(C_{I},e)$ . Then the ciphertext with the same $e$ can directly conduct operation op1 to get the encrypted summation of plaintext, ciphertext with different $e$ needed to recalibrate such that $e$ is the same. In practice, we use the same base exponent such that no information will leak from $e$ .

Negative number. A max number parameter is set to handle the negative numbers. The max number can be set to half of $n$ in pk, which means we assume all of our data is smaller than max number, such an assumption is easy to satisfy since $n$ is usually set to a very large prime number. Then we perform mode $n$ on all the plaintext, all the positive number have no changes and all the negative numbers become positive numbers greater than max number. In the decryption process, if the decrypted plaintext is greater than max number, we minus $n$ to get the correct negative plaintext.

FullText or PartText. Usually the rating or feedback comprises a sparse matrix Koren et al. (2009) which means the amount of feedback from a user could be very limited. Therefore, two different settings are implemented our system. Both of them follow the overall steps of FedMF, but are slightly different at the user uploading process. In one setting called FullText, users upload gradients for all the items; the gradient is set to 0 if a user does not rate an item. In the other setting called PartText, users only upload the gradients of the rated items. They both have advantages and disadvantages, PartText leaks information about which items the user has rated but has higher computation efficiency, FullText leaks no information but needs more computation time.

We utilize an open source python package, python-paillier333https://github.com/n1analytics/python-paillier to accomplish the encryption part in our prototype system.

6.2 Evaluation

Dataset: To test the feasibility of our system, we use a real movie rating dataset Harper and Konstan (2016) from MovieLens which contains 100K rating information made by 610 users on 9724 movies. This dataset is also used in other homomorphic-encrypted MF works such as Nikolaenko et al. (2013) and Kim et al. (2016).

Parameters: In Paillier encryption, we set the length of public key to 1024. The bandwidth of communication is set to 1 Gb/s. In the matrix factorization process, we set the dimension of user and item profile to 100.

Environment: All the test experiments are performed on a server with 5.0GHz 6-core CPU and 32GB RAM, where the operation system is Windows and the program language is Python. We used a module called gmpy 444gmpy is a c-coded Python extension module that supports multiple-precision arithmetic, https://github.com/aleaxit/gmpy to accelerate the homomorphic encryption part in Python such that it is as fast as C++ implementation.

Performance: Since neither distributed computing or homomorphic encryption mechanisms will affect the computation values, FedMF will output the same user and item profiles as the original MF algorithm. Hence, the major objective of the experiments is testing the computation time of FedMF. Fixing the number of users to 610, Table 1 shows the time consumption of each iteration of PartText and FullText (one iteration means all of the 610 users’ uploaded gradients are used to update the item profiles once). For both PartText and FullText, the time consumption is quite good when there are not too many items, and the time efficiency decreases when more items are given. Roughly, the time consumed for each iteration linearly increases with the number of items. Compared with Fulltext, PartText is more efficient but it leaks some information. Particularly, PartText is nearly 20 times faster than the Fulltext solution.

Fig. 2 and 3 show the ratio of the user and server updating time when the number of items changes. The communication time is dismissed from the figures because it is too small compared with the user and server updating time. For example, $\sim$ 80MB of gradient data need to be sent to server when the item number is 2560 and it will cost only 1.25 seconds. From these figures, we can find out that $\sim$ 95% of time in one iteration is spent on server updates, which means if we increase the computing power of the server or improve the homomorphic encryption method such that the complexity of computation on ciphertext is lowered, the time efficiency of the whole system will improve significantly. This would be our future work.

7 Conclusion and Future Work

In this paper, we propose a novel secure matrix factorization framework in federated machine learning, called FedMF. More specifically, we first prove that a distributed matrix factorization system where users send gradients to the server in forms of plaintext will leak users’ rating information. Then, we design a homomorphic encryption based secure matrix factorization framework. We have proved that our system is secure against an honest-but-curious server, and the accuracy is same as the matrix factorization on users’ raw data.

Experiments on real-world data show that FedMF’s time efficiency is acceptable when the number of items is small. Also note that our system’s time consumption linearly increases with the number of items. To make FedMF more practical in reality, we still face several challenges:

More efficient homomorphic encryption. As we have discussed before, about 95% of our system’s time consumption is spent on server updates, where the computation is all performed on the ciphertext. If we can improve the homomorphic encryption’s efficiency when conducting operations on ciphertext, our system’s performance will increase.

Between FullText and PartText. Our experiments have shown that PartText is much more efficient than FullText, but PartText reveals the set of items rated by a user. This information, without the exact rating scores, may still leak users’ sensitive information Yang et al. (2016). Perhaps we can ask users to upload more gradients than only the rated items, but not all the items, so as to increase efficiency compared to FullText, while not leaking the exactly rated item set.

More secure definitions. Currently, we use a typical horizontal federated learning secure definition, which assumes honest participants and an honest-but-curious server. Next, we can explore more challenging secure definitions, such as how to build a secure system where the server is honest-but-curious, and some participants are malicious and the malicious participants may collude with the server.

Bibliography16

The reference list from the paper itself. Each links out to its DOI / PubMed record.

1Acar et al. [2018] Abbas Acar, Hidayet Aksu, A Selcuk Uluagac, and Mauro Conti. A survey on homomorphic encryption schemes: Theory and implementation. ACM Computing Surveys (CSUR) , 51(4):79, 2018.
2Ammad-ud-din et al. [2019] Muhammad Ammad-ud-din, Elena Ivannikova, Suleiman A. Khan, Were Oyomno, Qiang Fu, Kuan Eeik Tan, and Adrian Flanagan. Federated collaborative filtering for privacy-preserving personalized recommendation system. Co RR , abs/1901.09888, 2019.
3Berlioz et al. [2015] Arnaud Berlioz, Arik Friedman, Mohamed Ali Kaafar, Roksana Boreli, and Shlomo Berkovsky. Applying differential privacy to matrix factorization. In Proceedings of the 9th ACM Conference on Recommender Systems , pages 107–114. ACM, 2015.
4Dwork [2011] Cynthia Dwork. Differential privacy. Encyclopedia of Cryptography and Security , pages 338–340, 2011.
5Goldreich [2009] Oded Goldreich. Foundations of cryptography: volume 2, basic applications . Cambridge university press, 2009.
6Harper and Konstan [2016] F Maxwell Harper and Joseph A Konstan. The movielens datasets: History and context. Acm transactions on interactive intelligent systems (tiis) , 5(4):19, 2016.
7Jager [2012] Tibor Jager. The generic composite residuosity problem. In Black-Box Models of Computation in Cryptology , pages 49–56. Springer, 2012.
8Kim et al. [2016] Sungwook Kim, Jinsu Kim, Dongyoung Koo, Yuna Kim, Hyunsoo Yoon, and Junbum Shin. Efficient privacy-preserving matrix factorization via fully homomorphic encryption. In Proceedings of the 11th ACM on Asia Conference on Computer and Communications Security , pages 617–628. ACM, 2016.