Secure Federated Transfer Learning

TL;DR

This paper introduces a secure federated transfer learning framework that enables multiple organizations to collaboratively improve machine learning models while preserving privacy, with minimal modifications and comparable accuracy to traditional methods.

Contribution

The paper proposes a novel federated transfer learning framework that enhances model performance across organizations without compromising privacy, including a secure validation method.

Findings

Achieves comparable accuracy to non-privacy-preserving models

Requires minimal modifications to existing models

Effective across various secure multi-party ML tasks

Abstract

Machine learning relies on the availability of a vast amount of data for training. However, in reality, most data are scattered across different organizations and cannot be easily integrated under many legal and practical constraints. In this paper, we introduce a new technique and framework, known as federated transfer learning (FTL), to improve statistical models under a data federation. The federation allows knowledge to be shared without compromising user privacy, and enables complimentary knowledge to be transferred in the network. As a result, a target-domain party can build more flexible and powerful models by leveraging rich labels from a source-domain party. A secure transfer cross validation approach is also proposed to guard the FTL performance under the federation. The framework requires minimal modifications to the existing model structure and provides the same level of accuracy as the non-privacy-preserving approach. This framework is very flexible and can be effectively adapted to various secure multi-party machine learning tasks.

Tables

Table 1. TABLE I: Comparison of weighted F1 scores.

Tasks	$N_c$	SST	TLT	TLL	LR	SVMs	SAEs
water vs. others	$100$	$0.698\pm 0.011$	$0.692\pm 0.062$	$0.691\pm 0.060$	$0.685\pm 0.020$	$0.640\pm 0.016$	$0.677\pm 0.048$
water vs. others	$200$	$0.707\pm 0.013$	$0.702\pm 0.010$	$0.701\pm 0.007$	$0.672\pm 0.023$	$0.643\pm 0.038$	$0.662\pm 0.010$
person vs. others	$100$	$0.703\pm 0.015$	$0.697\pm 0.010$	$0.697\pm 0.020$	$0.694\pm 0.026$	$0.619\pm 0.050$	$0.657\pm 0.030$
person vs. others	$200$	$0.735\pm 0.004$	$0.733\pm 0.009$	$0.735\pm 0.010$	$0.720\pm 0.004$	$0.706\pm 0.023$	$0.707\pm 0.008$
sky vs. others	$100$	$0.708\pm 0.015$	$0.700\pm 0.022$	$0.713\pm 0.006$	$0.694\pm 0.016$	$0.679\pm 0.018$	$0.667\pm 0.009$
sky vs. others	$200$	$0.724\pm 0.014$	$0.718\pm 0.033$	$0.718\pm 0.024$	$0.696\pm 0.026$	$0.680\pm 0.042$	$0.684\pm 0.056$

Table 2. TABLE II: Comparison of training time between SS and HE with the increasing dimension of hidden representation denoted by d 𝑑 d , the increasing number of target-domain features, and the increasing number of overlapping samples, respectively

	15	20	25	30	35	40
HE training time (sec)	29.12	41.03	53.88	66.74	81.91	101.02
SS training time (sec)	2.41	2.52	2.61	2.73	2.89	3.04
	5	10	15	20	25	30
HE training time (sec)	17.82	20.45	21.67	24.03	27.12	28.58
SS training time (sec)	2.28	2.35	2.39	2.43	2.46	2.48
	60	80	100	120	140
HE training time (sec)	74.21	89.91	111.12	123.79	146.48
SS training time (sec)	2.55	2.65	2.74	2.79	2.83

Equations

$$\underset{{\Theta }^A,{\Theta }^B}{\mathrm{argmin}}{\mathcal{L}}_1=\sum _{i=1}^{N_c}{\ell }_1(y_i^A,\phi (u_i^B))$$

$$\underset{{\Theta }^A,{\Theta }^B}{\mathrm{argmin}}{\mathcal{L}}_2=\sum _{i=1}^{N_{AB}}{\ell }_2(u_i^A,u_i^B)$$

$$\underset{{\Theta }^A,{\Theta }^B}{\mathrm{argmin}}\mathcal{L}={\mathcal{L}}_1+\gamma {\mathcal{L}}_2+\frac{\lambda }{2}({\mathcal{L}}_3^A+{\mathcal{L}}_3^B)$$

$$\frac{\partial \mathcal{L}}{\partial {\theta }_l^i}=\frac{\partial {\mathcal{L}}_1}{\partial {\theta }_l^i}+\gamma \frac{\partial {\mathcal{L}}_2}{\partial {\theta }_l^i}+\lambda {\theta }_l^i.$$

$${\ell }_1(y,\phi )\approx {\ell }_1(y,0)+\frac{1}{2}C(y)\phi +\frac{1}{8}D(y){\phi }^2$$

$$\frac{\partial {\ell }_1}{\partial \phi }=\frac{1}{2}C(y)+\frac{1}{4}D(y)\phi .$$

$$\begin{array}{rl}[[\mathcal{L}]]& =\sum _i^{N_c}([[{\ell }_1(y_i^A,0)]]+\frac{1}{2}C(y_i^A){\Phi }^A[[\mathcal{G}(u_i^B)]]\\ & +\frac{1}{8}D(y_i^A){\Phi }^A[[(\mathcal{G}(u_i^B){)}^{\prime }\mathcal{G}(u_i^B)]]({\Phi }^A{)}^{\prime })\\ & +\gamma \sum _i^{N_{AB}}([[{\ell }_2^B(u_i^B)]]+[[{\ell }_2^A(u_i^A)]]+\kappa u_i^A[[(u_i^B{)}^{\prime }]])\\ & +[[\frac{\lambda }{2}{\mathcal{L}}_3^A]]+[[\frac{\lambda }{2}{\mathcal{L}}_3^B]],\end{array}$$

$$\begin{array}{rl}[[\frac{\partial \mathcal{L}}{\partial {\theta }_l^B}]]& =\sum _i^{N_c}\frac{\partial (\mathcal{G}(u_i^B){)}^{\prime }\mathcal{G}(u_i^B)}{\partial u_i^B}[[(\frac{1}{8}D(y_i^A)({\Phi }^A{)}^{\prime }{\Phi }^A]]\frac{\partial u_i^B}{\partial {\theta }_l^B}\\ & +\sum _i^{N_c}[[\frac{1}{2}C(y_i^A){\Phi }^A]]\frac{\partial \mathcal{G}(u_i^B)}{\partial u_i^B}\frac{\partial u_i^B}{\partial {\theta }_l^B}\\ & +\sum _i^{N_{AB}}([[\gamma \kappa u_i^A]]\frac{\partial u_i^B}{\partial {\theta }_l^B}+[[\gamma \frac{\partial {\ell }_2^B(u_i^B)}{\partial {\theta }_l^B}]])+[[\lambda {\theta }_l^B]],\end{array}$$

$$\begin{array}{rl}[[\frac{\partial \mathcal{L}}{\partial {\theta }_l^A}]]& =\sum _j^{N_A}\sum _i^{N_c}(\frac{1}{4}D(y_i^A){\Phi }^A[[\mathcal{G}(u_i^B{)}^{\prime }\mathcal{G}(u_i^B)]]\\ & +\frac{1}{2}C(y_i^A)[[\mathcal{G}(u_i^B)]])\frac{\partial {\Phi }^A}{\partial u_j^A}\frac{\partial u_j^A}{\partial {\theta }_l^A}\\ & +\gamma \sum _i^{N_{AB}}([[\kappa u_i^B]]\frac{\partial u_i^A}{\partial {\theta }_l^A}+[[\frac{\partial {\ell }_2^A(u_i^A)}{\partial {\theta }_l^A}]])+[[\lambda {\theta }_l^A]].\end{array}$$

$$\begin{array}{rl}\mathcal{L}& =\sum _i^{N_C}{\ell }_1(y_i^A,0)+\frac{1}{2}C(y_i^A){\Phi }^A\mathcal{G}(u_i^B)\\ & +\left(\frac{1}{8}D(y_i^A){\Phi }^A\mathcal{G}(u_i^B)\right)\left({\Phi }^A\mathcal{G}(u_i^B)\right)\\ & +\gamma \sum _i^{N_{AB}}\left({\ell }_2^B(u_i^B)+{\ell }_2^A(u_i^A)+\kappa u_i^A(u_i^B{)}^{\prime }\right)\\ & +\frac{\lambda }{2}\left({\mathcal{L}}_3^A+{\mathcal{L}}_3^B\right)\end{array}$$

$$\begin{array}{rl}\frac{\partial \mathcal{L}}{\partial {\theta }_{\ell }^B}& =\sum _i^{N_C}\frac{1}{2}C(y_i^A){\Phi }^A\frac{\partial \mathcal{G}(u_i^B)}{\partial {\theta }_{\ell }^B}\\ & +2\left[\left(\frac{1}{8}D(y_1^A){\Phi }^A\mathcal{G}(u_i^B)\right)\left({\Phi }^A\frac{\partial \left(\mathcal{G}(u_i^B)\right)}{\partial {\theta }_{\ell }^B}\right)\right]\\ & +\sum _i^{N_{AB}}\left(\gamma \kappa u_i^A\frac{\partial u_i^B}{\partial {\theta }_{\ell }^B}+\gamma \frac{\partial {\ell }_2^B(u_i^B)}{\partial {\theta }_{\ell }^B}\right)+\lambda {\theta }_{\ell }^B\end{array}$$

$$\begin{array}{rl}\frac{\partial \mathcal{L}}{\partial {\theta }_l^A}& =\sum _i^{N_C}\frac{1}{2}C(y_i^A)\frac{\partial {\Phi }^A}{\partial {\theta }_{\ell }^A}\mathcal{G}(u_i^B)\\ & +2\left[\left(\frac{1}{8}D(y_i^A){\Phi }^A\mathcal{G}(u_i^B)\right)\left(\frac{\partial {\Phi }^A}{\partial {\theta }_{\ell }^A}\mathcal{G}(u_i^B)\right)\right]\\ & +\gamma \sum _i^{N_{AB}}\left(\kappa u_i^B\frac{\partial u_i^A}{\partial {\theta }_{\ell }^A}+\frac{\partial {\ell }_2^A(u_i^A)}{\partial {\theta }_{\ell }^A}\right)+\lambda {\theta }_{\ell }^A.\end{array}$$

$$\begin{array}{rl}{\mathcal{L}}_{AB}& =\frac{1}{2}\sum _i^{N_C}C(y_i^A){\Phi }^A\mathcal{G}(u_i^B)\\ & +\frac{1}{8}\left(D(y_i^A){\Phi }^A\mathcal{G}(u_i^B)\right)\left({\Phi }^A\mathcal{G}(u_i^B)\right)\\ & +\gamma \kappa \sum _i^{N_{AB}}{u}_i^A(u_i^B{)}^{\prime }\end{array}$$

$$\begin{array}{rl}\mathcal{L}& ={\mathcal{L}}_A+{\mathcal{L}}_B+\sum _i^{N_C}{\mathcal{L}}_{AB}^{(A,1)}(i){\mathcal{L}}_{AB}^{(B,1)}(i)\\ & +\left({\mathcal{L}}_{AB}^{(A,2)}(i){\mathcal{L}}_{AB}^{(B,1)}(i)\right)\left({\mathcal{L}}_{AB}^{(A,3)}(i){\mathcal{L}}_{AB}^{(B,1)}(i)\right)\\ & +\sum _i^{N_{AB}}{\mathcal{L}}_{AB}^{(A,4)}(i){\left({\mathcal{L}}_{AB}^{(B,2)}(i)\right)}^{\prime }.\end{array}$$

$$\begin{array}{rl}{D}_{AB}^{(B,\ell )}& =\sum _i^{N_C}\frac{1}{2}C(y_i^A){\Phi }^A\frac{\partial \mathcal{G}(u_i^B)}{\partial {\theta }_{\ell }^B}\\ & +2(\frac{1}{8}D(y_i^A){\Phi }^A)\mathcal{G}(u_i^B)({\Phi }^A)(\frac{\partial \mathcal{G}(u_i^B)}{\partial {\theta }_{\ell }^B})\\ & +\sum _i^{N_{AB}}\gamma \kappa u_i^A\frac{\partial u_i^B}{\partial {\theta }_{\ell }^B}\end{array}$$

$$\begin{array}{rl}\frac{\partial \mathcal{L}}{\partial {\theta }_{\ell }^B}& =D_B^{B,\ell }+\sum _i^{N_C}{\mathcal{L}}_{AB}^{(A,1)}(i)D_{AB,B,1}^{(B,\ell )}(i)\\ & +2\left({\mathcal{L}}_{AB}^{(A,2)}(i){\mathcal{L}}_{AB}^{(B,1)}(i)\right)\left({\mathcal{L}}_{AB}^{(A,3)}(i)D_{AB,B,1}^{(B,\ell )}(i)\right)\\ & +\sum _i^{N_{AB}}{\mathcal{L}}_{AB}^{(A,4)}(i)D_{AB,B,2}^{(B,\ell )}(i)\end{array}$$

$$\begin{array}{rl}{D}_{AB}^{(A,\ell )}& =\sum _i^{N_C}\frac{1}{2}C(y_i^A)\frac{\partial {\Phi }^A}{\partial {\theta }_{\ell }^A}\mathcal{G}(u_i^B)\\ & +2\left(\frac{1}{8}D(y_i^A){\Phi }^A\right)\mathcal{G}(u_i^B)\left(\frac{\partial {\Phi }^A}{\partial {\theta }_{\ell }^A}\right)\mathcal{G}(u_i^B)\\ & +\gamma \sum _i^{N_{AB}}\left(\kappa u_i^B\frac{\partial u_i^A}{\partial {\theta }_{\ell }^A}\right)\end{array}$$

$$\begin{array}{rl}\frac{\partial \mathcal{L}}{\partial {\theta }_{\ell }^A}& =D_A^{(A,\ell )}+\sum _i^{N_C}{D}_{AB,A,1}^{(A,\ell )}(i){\mathcal{L}}_{AB}^{(B,1)}(i)\\ & +2\left({\mathcal{L}}_{A,B}^{(A,2)}{\mathcal{L}}_{AB}^{(B,1)}(i)\right)\left(D_{AB,A,2}^{(A,\ell )}(i){\mathcal{L}}_{AB}^{(B,1)}(i)\right)\\ & +\gamma \sum _i^{N_{AB}}\left(D_{AB,A,3}^{(A,\ell }(i)(u_i^B{)}^{\prime }\right)\end{array}$$

Full text

††publicationid: pubid: Copyright (c) 2020 IEEE. Personal use is permitted. For any other purposes, permission must be obtained from the IEEE by emailing pubs-permissions@ieee.org.

1 Introduction

Recent Artificial Intelligence (AI) achievements have been depending on the availability of massive amounts of labeled data. For example, AlphaGo has been trained using a dataset containing 30 million moves from 160,000 actual games. The ImageNet dataset has over 14 million images. However, across various industries, most applications only have access to small or poor quality datasets. Labeling data is very expensive, especially in fields which require human expertise and domain knowledge. In addition, data needed for a specific task may not all be stored in one place. Many organizations may only have unlabeled data, and some other organizations may have very limited amounts of labels. It has been increasingly difficult from a legislative perspective for organizations to combine their data, too. For example, General Data Protection Regulation (GDPR) [1], a new bill introduced by the European Union, contains many terms that protect user privacy and prohibit organizations from exchanging data without explicit user approval. How to enable the large number of businesses and applications that have only small data (few samples and features) or weak supervision (few labels) to build effective and accurate AI models while complying with data privacy and security laws is a difficult challenge.

To address this challenge, Google introduced a federated learning (FL) system [2] in which a global machine learning model is updated by a federation of distributed participants while keeping their data stored locally. Their framework requires all contributors share the same feature space. On the other hand, secure machine learning with data partitioned in the feature space has also been studied [3]. These approaches are only applicable in the context of data with either common features or common samples under a federation. In reality, however, the set of such common entities may be small, making a federation less attractive and leaving the majority of the non-overlapping data under-utilized.

In this paper, we propose Federated Transfer Learning (FTL) to address the limitations of existing federated learning approaches. It leverages transfer learning [4] to provide solutions for the entire sample and feature space under a federation. Our contributions are as follows:

We formalize the research problem of federated transfer learning in a privacy-preserving setting to provide solutions for federation problems beyond the scope of existing federated learning approaches; 2. 2.

We provide an end-to-end solution to the proposed FTL problem and show that the performance of the proposed approach in terms of convergence and accuracy is comparable to non-privacy-preserving transfer learning; and 3. 3.

We provide some novel approaches to incorporate additively homomorphic encryption (HE) and secret sharing using beaver triples into two-party computation (2PC) with neural networks under the FTL framework such that only minimal modifications to the neural network is required and the accuracy is almost lossless.

2 Related Work

Recent years have witnessed a surge of studies on encrypted machine learning. For example, Google introduced a secure aggregation scheme to protect the privacy of aggregated user updates under their federarted learning framework [5]. CryptoNets [6] adapted neural network computations to work with data encrypted via Homomorphic Encryption (HE). SecureML [7] is a multi-party computing scheme which uses secret-sharing and Yao’s Garbled Circuit for encryption and supports collaborative training for linear regression, logistic regression and neural networks.

Transfer learning aims to build an effective model for an application with a small dataset or limited labels in a target domain by leveraging knowledge from a different but related source domain. In recent years, there have been tremendous progress in applying transfer learning to various fields such as image classification and sentiment analysis. The performance of transfer learning relies on how related the domains are. Intuitively, parties in the same data federation are usually organizations from the same industry. Therefore, they can benefit more from knowledge propagation. To the best of our knowledge, FTL is the first framework to enable federated learning to benefit from transfer learning.

3 Preliminaries and Security Definition

Consider a source domain dataset $\mathcal{D}_{A}:=\{(x_{i}^{A},y_{i}^{A})\}_{i=1}^{N_{A}}$ where $x_{i}^{A}\in R^{a}$ and $y_{i}^{A}\in\{+1,-1\}$ is the $i$ th label, and a target domain $\mathcal{D}_{B}:=\{x_{j}^{B}\}_{j=1}^{N_{B}}$ where $x_{j}^{B}\in R^{a}$. $\mathcal{D}_{A}$ and $\mathcal{D}_{B}$ are separately held by two private parties and cannot be exposed to each other legally. We assume that there exists a limited set of co-occurrence samples $\mathcal{D}_{AB}:=\{(x_{i}^{A},x_{i}^{B})\}_{i=1}^{N_{AB}}$ and a small set of labels for data from domain B in party A’s dataset: $\mathcal{D}_{c}:=\{(x_{i}^{B},y_{i}^{A})\}_{i=1}^{N_{c}}$, where $N_{c}$ is the number of available target labels. Without loss of generality, we assume all labels are in party A, but all the deduction here can be adapted to the case where labels exist in party B. One can find the set of commonly shared sample IDs in a privacy-preserving setting by masking data IDs with encryption techniques (e.g., the RSA scheme). We utilize the RSA Intersection module of the FATE111https://github.com/FederatedAI/FATE framework to align co-occurrence samples of the two parties. Given the above setting, the objective is for the two parities to build a transfer learning model to predict labels for the target-domain party accurately without exposing data to each other.

In this paper, we adopt a security definition in which all parties are honest-but-curious. We assume a threat model with a semi-honest adversary $\mathcal{D}$ who can corrupt at most one of the two data clients. For a protocol $P$ performing $(O_{A},O_{B})=P(I_{A},I_{B})$, where $O_{A}$ and $O_{B}$ are party A’s and party B’s outputs and $I_{A}$ and $I_{B}$ are their inputs, respectively, $P$ is secure against A if there exists an infinite number of $(I^{\prime}_{B},O^{\prime}_{B})$ pairs such that $(O_{A},O^{\prime}_{B})=P(I_{A},I^{\prime}_{B})$. It allows flexible control of information disclosure compared to complete zero knowledge security.

4 The Proposed Approach

In this section, we introduce our proposed transfer learning model.

Deep neural networks have been widely adopted in transfer learning to find implicit transfer mechanisms. Here, we explore a general scenario in which hidden representations of A and B are produced by two neural networks $u_{i}^{A}=\it{Net}^{A}(x_{i}^{A})$ and $u_{i}^{B}=\it{Net}^{B}(x_{i}^{B})$, where $u^{A}\in\mathbb{R}^{N_{A}\times d}$ and $u^{B}\in\mathbb{R}^{N_{B}\times d}$, and $\it{d}$ is the dimension of the hidden representation layer. The neural networks $\it{Net}^{A}$ and $\it{Net}^{B}$ serve as feature transformation functions that project the source features of party A and party B into a common feature subspace in which knowledge can be transferred between the two parties. Any other symmetric feature transformation techniques can be applied here to form the common feature subspace. However, neural networks can help us build an end-to-end solution to the proposed FTL problem.

To label the target domain, a general approach is to introduce a prediction function $\varphi(u_{j}^{B})=\varphi(u_{1}^{A},y_{1}^{A}...u_{N_{A}}^{A},y_{N_{A}}^{A},u_{j}^{B})$. Without losing much generality, we assume $\varphi(u_{j}^{B})$ is linearly separable. That is, $\varphi(u_{j}^{B})=\Phi^{A}\mathcal{G}(u_{j}^{B})$. In our experiment, we use a translator function, $\varphi(u_{j}^{B})=\frac{1}{N_{A}}\sum_{i}^{N_{A}}y_{i}^{A}u_{i}^{A}(u_{j}^{B})^{\prime}$, where $\Phi^{A}=\frac{1}{N_{A}}\sum_{i}^{N_{A}}y_{i}^{A}u_{i}^{A}$ and $\mathcal{G}(u_{j}^{B})=(u_{j}^{B})^{\prime}$. We can then write the training objective using the available labeled set as:

[TABLE]

where $\Theta^{A}$, $\Theta^{B}$ are training parameters of $\it{Net}^{A}$ and $\it{Net}^{B}$, respectively. Let $L_{A}$ and $L_{B}$ be the number of layers for $\it{Net}^{A}$ and $\it{Net}^{B}$, respectively. Then, $\Theta^{A}=\{\theta_{l}^{A}\}_{l=1}^{L_{A}}$, $\Theta^{B}=\{\theta_{l}^{B}\}_{l=1}^{L_{B}}$ where $\theta_{l}^{A}$ and $\theta_{l}^{B}$ are the training parameters for the $l$th layer. $\ell_{1}$ denotes the loss function. For logistic loss, $\ell_{1}(y,\varphi)=\log(1+\exp(-y\varphi))$.

In addition, we also aim to minimize the alignment loss between A and B in order to achieve feature transfer learning in a federated learning setting:

[TABLE]

where $\ell_{2}$ denotes the alignment loss. Typical alignment losses can be $-u_{i}^{A}(u_{i}^{B})^{\prime}$ or $||u_{i}^{A}-u_{i}^{B}||_{F}^{2}$. For simplicity, we assume that it can be expressed in the form of $\ell_{2}(u_{i}^{A},u_{i}^{B})=\ell_{2}^{A}(u_{i}^{A})+\ell_{2}^{B}(u_{i}^{B})+\kappa u_{i}^{A}(u_{i}^{B})^{\prime}$, where $\kappa$ is a constant.

The final objective function is:

[TABLE]

where $\gamma$ and $\lambda$ are the weight parameters, and $\mathcal{L}_{3}^{A}=\sum_{l}^{L_{A}}||\theta_{l}^{A}||_{F}^{2}$ and $\mathcal{L}_{3}^{B}=\sum_{l}^{L_{B}}||\theta_{l}^{B}||_{F}^{2}$ are the regularization terms. Now we focus on obtaining the gradients for updating $\Theta^{A}$, $\Theta^{B}$ in back propagation. For $i\in\{A,B\}$, we have:

[TABLE]

Under the assumption that A and B are not allowed to expose their raw data, a privacy-preserving approach needs to be developed to compute equations (3) and (4). Here, we adopt a second order Taylor approximation for logistic loss:

[TABLE]

and the gradient is:

[TABLE]

where, $C(y)=-y$, $D(y)=y^{2}$.

In the following two sections, we will discuss two alternative constructions of the secure FTL protocol: the first one is leveraging additively homomorphic encryption, and the second one is utilizing the secret sharing based on beaver triples. We carefully design the FTL protocol such that only minimal information needs to be encrypted or secretly shared between parties. Besides, the FTL protocol is designed to be compatible with other homomorphic encryption and secret sharing schemes with minimal modifications.

5 FTL using Homomorphic Encryption

Additively Homomorphic Encryption and polynomial approximations have been widely used for privacy-preserving machine learning. Applying equations (5) and (6), and additively homomorphic encryption (denoted as $[[\cdot]]$), we obtain the privacy preserved loss function and the corresponding gradients for the two domains as:

[TABLE]

[TABLE]

[TABLE]

5.1 FTL Algorithm - Homomorphic Encryption based

With equations (7), (8) and (9), we now design a federated algorithm for solving the transfer learning problem. See Figure 1. Let $[[\cdot]]_{A}$ and $[[\cdot]]_{B}$ be homomorphic encryption operators with public keys from A and B, respectively. A and B initialize and execute their respective neural networks $\it{Net^{A}}$ and $\it{Net^{B}}$ locally to obtain hidden representations ${u_{i}^{A}}$ and ${u_{i}^{B}}$. A then computes and encrypts components $\{\it{h}_{k}(u_{i}^{A},y_{i}^{A})\}_{k=1,2...K_{A}}$ and sends to B to assist calculations of gradients of $\it{Net^{B}}$. In the current scenario, $K_{A}=3$, $\it{h}_{1}^{A}(u_{i}^{A},y_{i}^{A})=\{[[\frac{1}{8}D(y_{i}^{A})(\Phi^{A})^{\prime}(\Phi^{A})]]_{A}\}_{i=1}^{N_{c}}$, $\it{h}_{2}^{A}(u_{i}^{A},y_{i}^{A})=\{[[\frac{1}{2}C(y_{i}^{A})\Phi^{A}]]_{A}\}_{i=1}^{N_{c}}$, and $\it{h}_{3}^{A}(u_{i}^{A},y_{i}^{A})=\{[[\gamma\kappa u_{i}^{A}]]_{A}\}_{i=1}^{N_{AB}}$. Similarly, B then computes and encrypts components $\{\it{h}_{k}^{B}(u_{i}^{B})\}_{k=1,2...K_{B}}$ and sends to A to assist calculations of gradients of $\it{Net^{A}}$ and loss $\mathcal{L}$. In the current scenario, $K_{B}=4$, $\it{h}_{1}^{B}(u_{i}^{B})=\{[[(\mathcal{G}(u_{i}^{B}))^{\prime}\mathcal{G}(u_{i}^{B})]]_{B}\}_{i=1}^{N_{c}}$, $\it{h}_{2}^{B}(u_{i}^{B})=\{[[\mathcal{G}(u_{i}^{B})]]_{B}\}_{i=1}^{N_{c}}$, $\it{h}_{3}^{B}(u_{i}^{B})=\{[[\kappa u_{i}^{B}]]_{B}\}_{i=1}^{N_{AB}}$, and $\it{h}_{4}^{B}(u_{i}^{B})=[[\frac{\lambda}{2}\mathcal{L}_{3}^{B}]]_{B}$.

To prevent A’s and B’s gradients from being exposed, A and B further mask each gradient with an encrypted random value. They then send the masked gradients and loss to each other and decrypt the values locally. A can send termination signals to B once the loss convergence condition is met. Otherwise, they unmask the gradients, update the weight parameters with their respective gradients, and move on to next iteration. Once the model is trained, FTL can provide predictions for unlabeled data from party B. Algorithm 1 summaries the prediction process.

5.2 Security Analysis

Theorem 1**.**

The protocol in the FTL training Algorithm (Figure 1) and Algorithm 1 is secure under our security definition, provided that the underlying additively homomorphic encryption scheme is secure.

Proof.

The training protocol in Figure 1 and Algorithm 1 do not reveal any information, because all A and B learned are the masked gradients. In each iteration, A and B create new random masks. The strong randomness and secrecy of the masks secure the information against the other party [8]. During training, A learns its own gradients at each step, but this is not enough for A to learn any information from B based on the impossibility of solving $n$ equations with more than $n$ unknowns [8]. In other words, there exist an infinite number of inputs from B that result in the same gradients A receives. Similarly, B cannot learn any information from A. Therefore, as long as the encryption scheme is secure, the proposed protocol is secure. During evaluation, A learns the predicted result for each sample from B, which is a scalar product, from which A cannot infer B’s private information. B learns only the label, from which B cannot infer A’s private information. ∎

At the end of the training process, each party (A or B) only obtains the model parameters associated with its own features, and remains oblivious to the data structure of the other party. At inference time, the two parties need to collaboratively compute the prediction results. Note that the protocol does not deal with the situation in which one (or both) of the two parties is (are) malicious. If A fakes its inputs and submits only one non-zero input, it might be able to tell the value of $u_{i}^{B}$ at the position of that input. It still cannot tell $x_{i}^{B}$ or $\Theta_{B}$, and neither party can obtain the correct prediction results.

6 FTL using secret sharing

Throughout this section, assume that any private value $v$ is shared between the two parties where $A$ keeps $\langle v\rangle_{A}$ and $B$ keeps $\langle v\rangle_{B}$ such that $v=\langle v\rangle_{A}+\langle v\rangle_{B}.$ To make it possible for the performance to be comparable with the previous construction, assume that $\ell_{1}(y,\varphi)$ and $\frac{\partial\ell_{1}}{\partial\varphi}$ can be approximated by the second order Taylor expansion following Equations (5) and (6). So $\mathcal{L},\frac{\partial\mathcal{L}}{\partial\theta_{\ell}^{A}}$ and $\frac{\partial\mathcal{L}}{\partial\theta_{\ell}^{B}}$ can be expressed as the following

[TABLE]

[TABLE]

[TABLE]

In this case, the whole process can be performed securely if secure matrix addition and multiplication can be constructed. Since operations with public matrices or adding two private matrices can simply be done using the shares without any communication, the remaining operation that requires discussion is secure matrix multiplication. Beaver’s triples are used to help in the matrix multiplication.

6.1 Secure Matrix Multiplication using Beaver Triples

First, we briefly recall how to perform the matrix multiplication given that the two parties have already shared a Beaver’s triple. Suppose that the computation required is to obtain $P=MN$ where the dimensions of $M,N$ and $P$ are $m\times n,n\times k$ and $m\times k$ respectively. As assumed, the matrices $M$ and $N$ have been secretly shared by the two parties where $A$ keeps $\langle M\rangle_{A}$ and $\langle N\rangle_{A}$ and $B$ keeps $\langle M\rangle_{B}$ and $\langle N\rangle_{B}.$ To assist with the calculation, in the preprocessing phase, $A$ and $B$ have generated three matrices $D,E,F$ of dimension $m\times n,n\times k$ and $m\times k$ respectively where $A$ keeps $\langle D\rangle_{A},\langle E\rangle_{A}$ and $\langle F\rangle_{A}$ while $B$ keeps $\langle D\rangle_{B},\langle E\rangle_{B}$ and $\langle F\rangle_{B}$ such that $DE=F.$

It is easy to see that $\langle P\rangle_{A}+\langle P\rangle_{B}=MN,$ which is what is required for this protocol. As can be seen, this method guarantees efficient online computation in the cost of having offline phase where players generated the Beavers triples. So next we discuss the scheme to generate the triples.

6.2 Beaver Triples Generation

In the preprocessing phase, the Beaver’s triples generation protocol uses a sub-protocol to perform the secure matrix multiplication with the help of a third party, which we will call Carlos. Recall that having two matrices $U$ and $V$ owned respectively by $A$ and $B,$ they want to calculate $UV$ securely with the help of Carlos. In order to do this, Alice and Bob individually generate shares for $U$ and $V$ respectively. That is, we have $U=\langle U\rangle_{0}+\langle U\rangle_{1}$ and $V=\langle V\rangle_{0}+\langle V\rangle_{1}.$ Then we have $UV=(\langle U\rangle_{0}+\langle U\rangle_{1})\cdot(\langle V\rangle_{0}+\langle V\rangle_{1})=\langle U\rangle_{0}\langle V\rangle_{0}+\langle U\rangle_{0}\langle V\rangle_{1}+\langle U\rangle_{1}\langle V\rangle_{0}+\langle U\rangle_{1}\langle V\rangle_{1}.$ So if Alice sends $\langle U\rangle_{1}$ to Bob and Bob sends $\langle V\rangle_{0}$ to Alice:

$\langle U\rangle_{0}\langle V\rangle_{0}$ can be privately calculated by Alice 2. 2.

$\langle U\rangle_{1}\langle V\rangle_{1}$ can be privately calculated by Bob 3. 3.

$\langle U\rangle_{1}\langle V\rangle_{0}$ can be privately calculated by both Alice and Bob 4. 4.

However, no one can calculate $\langle U\rangle_{0}\langle V\rangle_{1}$ yet. This is what Carlos will calculate.

By the use of Algorithm 3, Algorithm 4 generates triple $(D,E,F)$ such that:

$DE=F$ 2. 2.

Alice holds $\langle D\rangle_{A},\langle E\rangle_{A}$ and $\langle F\rangle_{A}$ without learning anything about $(D,E,F),\langle D\rangle_{B},\langle E\rangle_{B}$ and $\langle F\rangle_{B}.$ 3. 3.

Bob holds $\langle D\rangle_{B},\langle E\rangle_{B}$ and $\langle F\rangle_{B}$ without learning anything about $(D,E,F),\langle D\rangle_{A},\langle E\rangle_{A}$ and $\langle F\rangle_{A}.$

Lastly, during offline phase, Alice and Bob also requested Carlos to generate sufficient number of shares for zero matrices with various dimensions.

6.3 FTL Algorithm - Secret Sharing based

Before discussing our FTL protocol that is constructed based on Beaver triples, we first give some notation to simplify Equations (10), (11) and (12) based on the parties needed to complete the calculation.

Let $\mathcal{L}_{A}=\sum_{i}^{N_{C}}\ell_{1}(y_{i}^{A},0)+\gamma\sum_{i}^{N_{AB}}\ell_{2}^{A}(u_{i}^{A})+\frac{\lambda}{2}\mathcal{L}_{3}^{A},$ 2. 2.

Let $\mathcal{L}_{B}=\gamma\sum_{i}^{N_{AB}}\ell_{2}^{B}(u_{i}^{B})+\frac{\lambda}{2}(\mathcal{L}_{3}^{B}),$ 3. 3.

Let

[TABLE]

with

• •

For $A:$

• –

$\mathcal{L}_{AB}^{(A,1)}=\left(\frac{1}{2}C(y_{i}^{A})\Phi^{A}\right)_{i=1,\cdots,N_{C}},$

• –

$\mathcal{L}_{AB}^{(A,2)}=\left(\frac{1}{8}D(y_{i}^{A})\Phi^{A}\right)_{i=1,\cdots,N_{C}}$

• –

$\mathcal{L}_{AB}^{(A,3)}=\left(\Phi^{A}\right)_{i=1,\cdots,N_{C}}$ and

• –

$\mathcal{L}_{AB}^{(A,4)}=(\gamma\kappa u_{i}^{A})_{i=1,\cdots,N_{AB}}.$

• •

For $B:$

• –

$\mathcal{L}_{AB}^{(B,1)}=\left(\mathcal{G}(u_{i}^{B})\right)_{i=1,\cdots,N_{C}},$ and

• –

$\mathcal{L}_{AB}^{(B,2)}=(u_{i}^{B})_{i=1,\cdots,N_{C}}.$

Then

[TABLE] 4. 4.

Let $D^{(B,\ell)}_{B}=\sum_{i}^{N_{AB}}\gamma\frac{\partial\ell_{2}^{B}(u_{i}^{B})}{\partial\theta_{\ell}^{B}}+\lambda\theta_{\ell}^{B},$ 5. 5.

Let

[TABLE]

with

• •

For $B$:

• –

$D_{AB,B,1}^{(B,\ell)}=\left(\frac{\partial\mathcal{G}(u_{i}^{B})}{\partial\theta_{\ell}^{B}}\right)_{i=1,\cdots,N_{C}},$

• –

$D_{AB,B,2}^{(B,\ell)}=\left(\frac{\partial u_{i}^{B}}{\partial\theta_{\ell}^{B}}\right)_{i=1,\cdots,N_{AB}}.$

Then

[TABLE] 6. 6.

Let $D^{(A,\ell)}_{A}=\sum_{i}^{N_{AB}}\gamma\frac{\partial\ell_{2}^{A}(u_{i}^{A})}{\partial\theta_{\ell}^{A}}+\lambda\theta_{\ell}^{A},$ 7. 7.

Let

[TABLE]

with

• •

For $A:$

• –

$D_{AB,A,1}^{(A,\ell)}=\left(\frac{1}{2}C(y_{i}^{A})\frac{\partial\Phi^{A}}{\partial\theta_{\ell}^{A}}\right)_{i=1,\cdots,N_{C}},$

• –

$D_{AB,A,2}^{(A,\ell)}=\left(\frac{\partial\Phi^{A}}{\partial\theta_{\ell}^{A}}\right)_{i=1,\cdots,N_{C}}$ and

• –

$D_{AB,A,3}^{(A,\ell)}=\left(\gamma\kappa\frac{\partial u_{i}^{A}}{\partial\theta_{\ell}^{A}}\right)_{i=1,\cdots,N_{AB}}.$

Then

[TABLE]

To perform the training scheme, both Alice and Bob first initialize and execute their respective neural networks $Net^{A}$ and $Net^{B}$ locally to obtain $u_{i}^{A}$ and $u_{i}^{B}.$ Alice computes $\{h_{k}^{A}(u_{i}^{A},y_{i}^{A})\}_{k=1,\cdots,K_{A}}.$ Then for each $k,$ she randomly chooses a mask $\langle h_{k}^{A}(u_{i}^{A},y_{i}^{A})\rangle_{A}$ and sets $\langle h_{k}^{A}(u_{i}^{A},y_{i}^{A})\rangle_{B}=h_{k}^{A}(u_{i}^{A},y_{i}^{A})-\langle h_{k}^{A}(u_{i}^{A},y_{i}^{A})\rangle_{A}.$ Then Alice sends $\langle h_{k}^{A}(u_{i}^{A},y_{i}^{A})\rangle_{B}$ to Bob for $k=1,\cdots,K_{A}.$ Similarly, Bob computes $\{h_{k}^{B}(u_{i}^{B})\}_{k=1,\cdots,K_{B}}$ and for each $k,$ he randomly chooses $\langle h_{k}^{B}(u_{i}^{B})\rangle_{B}$ and sets $\langle h_{k}^{B}(u_{i}^{B})\rangle_{A}=h_{k}^{B}(u_{i}^{B})-\langle h_{k}^{B}(u_{i}^{B})\rangle_{B}$, which is then sent to Alice.

In our scenario, $K_{A}=7,$ with:

$h_{1}^{A}(u_{i}^{A},y_{i}^{A})=\mathcal{L}_{AB}^{(A,1)},$ 2. 2.

$h_{2}^{A}(u_{i}^{A},y_{i}^{A})=\mathcal{L}_{AB}^{(A,2)},$ 3. 3.

$h_{3}^{A}(u_{i}^{A},y_{i}^{A})=\mathcal{L}_{AB}^{(A,3)},$ 4. 4.

$h_{4}^{A}(u_{i}^{A},y_{i}^{A})=\mathcal{L}_{AB}^{(A,4)},$ 5. 5.

$h_{5}^{A}(u_{i}^{A},y_{i}^{A})=\mathcal{D}_{AB,A,1}^{(A,\ell)},$ 6. 6.

$h_{6}^{A}(u_{i}^{A},y_{i}^{A})=\mathcal{D}_{AB,A,2}^{(A,\ell)},$ 7. 7.

$h_{7}^{A}(u_{i}^{A},y_{i}^{A})=\mathcal{D}_{AB,A,3}^{(A,\ell)}$

and $K_{B}=4$ with

$h_{1}^{B}(u_{i}^{B})=\mathcal{L}_{AB}^{(B,1)},$ 2. 2.

$h_{2}^{B}(u_{i}^{B})=\mathcal{L}_{AB}^{(B,2)},$ 3. 3.

$h_{3}^{B}(u_{i}^{B})=\mathcal{D}_{AB,B,1}^{(B,\ell)}$ 4. 4.

$h_{4}^{B}(u_{i}^{B})=\mathcal{D}_{AB,B,2}^{(B,\ell)}.$

In addition, Alice privately computes $\mathcal{L}_{A}$ and $D_{A}^{(A,\ell)}$ while Bob privately computes $\mathcal{L}_{B}$ and $D_{B}^{(B,\ell)}.$ Algorithm 6 provides the training protocol for one iteration based on Beaver triples generated by Algorithm 4.

After the training is completed, we proceed to the prediction phase. Recall that after the training phase, Alice has the optimal value for $\Theta^{A}$ while Bob has the optimal value for $\Theta^{B}.$ Suppose that now $B$ wants to learn the label for $\{x_{j}^{B}\}_{j\in N_{B}}.$ The protocol can be found in Algorithm 6.

Theorem 2**.**

The protocol in Algorithms 3,4,5 and 6 are information theoretically secure against at most one passive adversary.

Proof.

Note that in all of these algorithms, the only information that any party receives regarding any private values is only the share for an $n$-out-of-$n$ secret sharing scheme. So by the property of $n$-out-of-$n$ secret sharing scheme, no one can learn any information about the private values they are not supposed to learn. After the calculation, the same thing can be said since each party only learns about a share of a secret sharing scheme and they cannot learn any information regarding values they are not supposed to learn from there. ∎

Remark 1**.**

Using the argument in [7] we can improve the efficiency in the following manner; for each matrix $A,$ it is always masked by the same random matrix. This optimization does not affect the security of the protocol while significantly improves the efficiency.

7 Experimental Evaluation

In this section, we report experiments conducted on public datasets including: 1) NUS-WIDE dataset [9] 2) Kaggle’s Default-of-Credit-Card-Clients [10] (“Default-Credit”) to validate our proposed approach. We study the effectiveness and scalability of the approach with respect to various key factors, including the number of overlapping samples, the dimension of hidden common representations, and the number of features.

The NUS-WIDE dataset consists of 634 low-level features from Flickr images as well as their associated tags and ground truth labels. There are in total 81 ground truth labels. We use the top 1,000 tags as text features and combine all the low-level features including color histograms and color correlograms as image features. We consider solving a one-vs-all classification problem with a data federation formed between party A and party B, where A has 1000 text tag features and labels, while party B has 634 low-level image features.

The “Default-Credit” dataset consists of credit card records including user demographics, history of payments, and bill statements, etc., with users’ default payments as labels. After applying one-hot encoding to categorical features, we obtain a dataset with 33 features and 30,000 samples. We then split the dataset both in the feature space and the sample space to simulate a two-party federation problem. Specifically, we assign each sample to party A, party B or both so that there exists a small number of samples overlapping between A and B. All labels are on the side of party A. We will examine the scalability (in section 7.3) of the FTL algorithm by dynamically splitting the feature space.

7.1 Impact of Taylor Approximation

We studied the effect of Taylor approximation by monitoring and comparing the training loss decay and the performance of prediction. Here, we test the convergence and precision of the algorithm using the NUS-WIDE data and neural networks with different levels of depth. In the first case, $Net^{A}$ and $Net^{B}$ both have one auto-encoder layer with 64 neurons, respectively. In the second case, $Net^{A}$ and $Net^{B}$ both have two auto-encoder layers with 128 and 64 neurons, respectively. In both cases, we used 500 training samples, 1,396 overlapping pairs, and set $\gamma=0.05,\lambda=0.005$. We summarize the results in Figures 2(a) and 2(b). We found that the loss decays at a similar rate when using Taylor approximation as compared to using the full logistic loss, and the weighted F1 score of the Taylor approximation approach is also comparable to the full logistic approach. The loss converges to a different minima in each of these cases. As we increased the depth of the neural networks, the convergence and the performance of the model did not decay.

Most existing secure deep learning frameworks suffer from accuracy loss when adopting privacy-preserving techniques [7]. Using only low-degree Taylor approximation, the drop in accuracy in our approach is much less than the state-of-art secure neural networks with similarly approximations.

7.2 Performance

We tested SS-based FTL (SST), HE-based FTL with Taylor loss (TLT) and FTL with logistic loss (TLL). For the self-learning approach, we picked three machine learning models: 1) logistic regression (LR), 2) SVM, and 3) stacked auto-encoders (SAEs). The SAEs are of the same structure as the ones we used for transfer learning, and are connected to a logistic layer for classification. We picked three of the most frequently occurring labels in the NUS-WIDE dataset, i.e., water, person and sky, to conduct one vs. others binary classification tasks. For each experiment, the number of overlapping samples we used is half of the total number of samples in that category. We varied the size of the training sample set and conducted three tests for each experiment with different random partitions of the samples. The parameters $\lambda$ and $\gamma$ are optimized via cross-validation.

Figure 2(c) shows the effect of varying the number of overlapping samples on the performance of transfer learning. The overlapping sample pairs are used to bridge the hidden representations between the two parties. The performance of FTL improves as the overlap between datasets increases.

The comparison of F-score (mean $\pm$ std) among SST, TLT, TLL and the several other machine learning models is shown in Table I. We observe that SST, TLT and TLL yield comparable performance across all tests. This demonstrates that SST can achieve plain-text level accuracy while TLT can achieve almost lossless accuracy although Taylor approximation is applied. The three FTL models outperform baseline self-learning models significantly using only a small set of training samples under all experimental conditions. In addition, performance improves as we increased the number of training samples. The results demonstrated the robustness of FTL.

7.3 Scalability

We study the scalability using Default-Credit dataset because it allows us to conveniently choose features when we do experiments. Specifically, we study how the training time scales with the number of overlapping samples, the number of target-domain features, and the dimension of hidden representations, denote as $d$. Based on the algorithmic detail of proposed transfer learning approach, the communication cost for B sending a message to A can be calculated by formula $Cost_{B\xrightarrow{}A}=n*(d^{2}+d)*ct$, where $ct$ is the size of the message and $n$ is the number of samples sent. The same cost applies when sending message from A to B.

To speed up the secure FTL algorithm, we preform compute-intensive operations in parallel. The logic flow of parallel secure FTL algorithm includes three stages: parallel encryption, parallel gradient calculation, and parallel decryption. Detailed logic flow is shown in Figure 3.

On parallel encryption stage, we parallelly encrypt components that will be sent to the other party. On parallel gradient calculation stage, we parallelly perform operations, including matrix multiplication and addition, on encrypted components to calculate encrypted gradients. On parallel decryption stage, we parallelly decrypt masked loss and gradients. Finally, the two parties exchange decrypted masked gradients that will be used to update neural networks. With 20 partitions, the parallel scheme can boost the secure FTL 100x than sequential scheme.

Figures 2(d), 2(e) and 2(f) illustrate that with parallelism applied, the running time of HE-based FTL grows approximately linearly with respect to the size of the hidden representation dimension, the number of target-domain features, as well as the number of overlapping samples respectively.

Figures 2(g), 2(h) and 2(i) illustrate how the training time varies with the three key factors in the SS setting. The communication cost can be simplified as $O(d^{2})$ if keeping other factors constant. As illustrated in Figures 2(g), however, the increasing rate of the training time is approaching linear rather than $O(d^{2})$. We conjecture that this is due to the computational efficiency of SS-based FTL. Besides, as illustrated in Figure 2(h) and 2(i), respectively, as the feature sizes or overlapping samples increase, the increasing rate of training time drops.

Further, we compare the scalability of SS-based with that of HE-based FTL along the axis of the hidden representation dimension, the number of features, and the number of overlapping samples, respectively. The results are presented in Table II. We notice that SS-based FTL is running much faster than HE-based FTL. Overall, SS-based FTL speeds up by 1-2 orders of magnitude compared with HE-based FTL. In addition, as shown in the three tables, the increasing rate of the training time of SS-based FTL is much slower than that of HE-based FTL.

8 Conclusions and Future Work

In this paper we proposed a secure Federated Transfer Learning (FTL) framework to expand the scope of existing secure federated learning to broader real-world applications. Two secure approaches, namely, homomorphic encryption (HE) and secret sharing are proposed in this paper for preserving privacy. The HE approach is simple, but computationally expensive. The biggest advantages of the secret sharing approach include (i) there is no accuracy loss, (ii) computation is much faster than HE approach. The major drawback of the secret sharing approach is that one has to offline generate and store many triplets before online computation.

We demonstrated that, in contrast to existing secure deep learning approaches which suffer from accuracy loss, FTL is as accurate as non-privacy-preserving approaches, and is superior to non-federated self-learning approaches. The proposed framework is a general privacy-preserving federated transfer learning solution that is not restricted to specific models.

In future research, we will continue improving the efficiency of the FTL framework by using distributed computing techniques with less expensive computation and communication schemes.