Proof-of-Contribution-Based Design for Collaborative Machine Learning on Blockchain

TL;DR

This paper presents a blockchain-based data marketplace for decentralized federated learning that ensures fair contribution-based rewards, privacy, robustness, verifiability, and efficiency.

Contribution

It introduces a novel blockchain marketplace design incorporating proof-of-contribution, privacy preservation, robustness, and verifiability using zero-knowledge proofs, with practical implementation and experiments.

Findings

Successful implementation of the proposed blockchain marketplace.

Effective contribution assessment and outlier detection demonstrated.

Fair reward distribution aligned with contributions verified through zero-knowledge proofs.

Abstract

We consider a project (model) owner that would like to train a model by utilizing the local private data and compute power of interested data owners, i.e., trainers. Our goal is to design a data marketplace for such decentralized collaborative/federated learning applications that simultaneously provides i) proof-of-contribution based reward allocation so that the trainers are compensated based on their contributions to the trained model; ii) privacy-preserving decentralized model training by avoiding any data movement from data owners; iii) robustness against malicious parties (e.g., trainers aiming to poison the model); iv) verifiability in the sense that the integrity, i.e., correctness, of all computations in the data market protocol including contribution assessment and outlier detection are verifiable through zero-knowledge proofs; and v) efficient and universal design. We propose a blockchain-based marketplace design to achieve all five objectives mentioned above. In our design, we utilize a distributed storage infrastructure and an aggregator aside from the project owner and the trainers. The aggregator is a processing node that performs certain computations, including assessing trainer contributions, removing outliers, and updating hyper-parameters. We execute the proposed data market through a blockchain smart contract. The deployed smart contract ensures that the project owner cannot evade payment, and honest trainers are rewarded based on their contributions at the end of training. Finally, we implement the building blocks of the proposed data market and demonstrate their applicability in practical scenarios through extensive experiments.

Tables

Table 1. TABLE I: Experimental results for the Outlier Detection

Attack	Model	Cross-round accuracy	Cross-trainer accuracy
Byzantine	LR	100%	100%
Byzantine	Resnet56	100%	100%
Backdoor	LR	100%	90%
Backdoor	Resnet56	100%	100%

Table 2. TABLE II: Model Inference Cost In ZKP

Model	Circuit Size	Proving (s)	Verification (ms)
LR	230,440	4.21	3
MLP	1,629,932	22.375	3
DLRM	31,860,124	401	3

Table 3. TABLE III: Cost of the Entire ZKP System. ∗ denotes estimation.

System	Circuit Size	Proving (s)	Verification (ms)
LR	5,605,226	75.364	3
MLP	43,974,762	591	3
DLRM	661,118,783	2.5h∗	3

Equations

$$L(\mathbf{w})=\sum _{i=1}^L\frac{n_i}{n}{L}_i(\mathbf{w}),$$

$$C_i^{(t)}=\text{Acc}\left({\mathbf{w}}^{(t)}\right)-\text{Acc}\left({\mathbf{w}}_{-i}^{(t)}\right),$$

$$S_c({\mathbf{w}}_i^{(t)},{\mathbf{w}}_i^{(t+1)})=\frac{{\mathbf{w}}_i^{(t)}\cdot {\mathbf{w}}_i^{(t+1)}}{\Vert {\mathbf{w}}_i^{(t)}\Vert \Vert {\mathbf{w}}_i^{(t+1)}\Vert }.$$

$$\mu =\frac{{\sum }_{\ell \in {\mathcal{S}}_E}\ell }{|{\mathcal{S}}_E|},\sigma =\sqrt{\frac{{\sum }_{\ell \in {\mathcal{S}}_E}(\ell -\mu {)}^2}{|{\mathcal{S}}_E|-1}}.$$

$$R_i=R_p+\alpha \frac{{\bar{C}}_i}{{\sum }_{j\in [L^{\prime }]}{\bar{C}}_j},$$

Full text

I Introduction

Over the past decade or so, our lives have transitioned into a fully connected reality where our smartphones, wearables, and other smart devices are constantly able to collect information about us and our surroundings. With this amount of available data, more-informed decisions (whether by smart assistants, service providers, vendors or government officials) can be made by leveraging machine learning (ML), in particular training of deep neural networks (DNNs). For example, real-time location data from driver devices can help route traffic around the city, improving the congestion situation in rush hours; vendors can improve their advertisement as well as their line-up of offerings based on analyzing information from collective information from their clients’ behaviors. Even though an enterprise may only be interested in learning useful aggregate information about its user population, data privacy has become an increasingly precious and critical commodity that individuals and/or enterprises are suspicious to give up, even for a greater-good goal. To deal with these challenges, collaborative/federated learning (FL) has been introduced to get benefit from outsourcing the computations as well as using participants’ local datasets to learn a more accurate model [1].

Federated learning (FL). As shown in Fig. 1, an FL system consists of a central server (aggregator) and a certain number of data owners (trainers) that collaboratively train a global model under the orchestration of the aggregator. Each data owner $i$ has a private dataset $D_{i}=\{(x_{1},y_{1}),\dots,(x_{n_{i}},y_{n_{i}})\}$ of size $n_{i}$, $i\in[L]$. The goal of the aggregator is to minimize

[TABLE]

where $L_{i}(\mathbf{w})=\frac{1}{n_{i}}\sum_{(x,y)\in\mathcal{D}_{i}}\ell(\mathbf{w};(x,y))$ is a predefined empirical local loss function over the dataset $\mathcal{D}_{i}$.

In each round $t$, the aggregator sends the updated global model $\mathbf{w}{(t)}$ to the trainers. Each trainer $i$ computes gradient vector $\nabla_{\mathbf{w}}L_{i}(\mathbf{w})$ over its private dataset and computes a local model $\mathbf{w}_{i}(t)$. When the data owners have equal-sized datasets, the aggregator computes an updated global model with $\mathbf{w}{(t+1)}=\frac{1}{L}\sum_{i\in[L]}\mathbf{w}_{i}(t).$ Upon aggregation, the aggregator pushes $\mathbf{w}(t+1)$ back to the trainers and this iterative process continues until the model converges.

Over the past few years, there has been a surge of research in FL, in particular on algorithmic developments to address data heterogeneity [2], personalization [3], and fairness [4]; privacy enhancements to address potential data leakage from locally trained models of the participants [5, 6, 7, 8] and security enhancements to address potential attacks in FL systems [9], as well as applications of FL to various ML domains [10].

Despite all these advances and benefits, collaborative ML systems (including FL) still need to provide sufficient incentive mechanisms to the participants. This is particularly important with the surge of Web3 era and rising interests for content and data monetization. To that end, data marketplaces can be utilized to incentivize interested parties to participate in the learning algorithm, use their local data, and perform heavy computations in exchange for monetary compensation.

In this work, our goal is to design a blockchain-empowered data marketplace for collaborative learning that is executed through a smart contract. We have a project (model) owner that wants to train a model by utilizing the compute power and private datasets of data owners (trainers)111Throughout the text, we use terms project owner and model owner interchangeably. Similarly, we use terms data owner and trainer interchangeably.. Key features of our design are proof-of-contribution-based fair reward allocation to trainers according to their actual contributions to the model; privacy-preserving training such that private local datasets of the trainers never leave their devices; verifiability such that the integrity of the entire data market operation is verifiable; robustness against malicious parties; and efficient implementation and universal applicability to different learning tasks. We utilize a decentralized storage infrastructure so that large ML datasets and models are stored off-chain. Heavy computations such as model aggregation, outlier detection, and contribution assessment are performed off-chain by a processing node, which we call the aggregator. The use of blockchain guarantees that the project owner cannot evade payment.

We propose a new metric called the randomized leave-one-out (RLOO) to assess the contribution of each trainer to the model. To detect malicious trainers, we propose a novel outlier detection scheme that is inspired by the three sigma rule [11] and the Krum defense [12]. To make the entire data market operation verifiable, we use zero-knowledge proofs (ZKPs) for model aggregation, outlier detection, and contribution assessment, which are generated by the aggregator and verified by the smart contract. For ZKPs, we use the Groth16 [13] zkSNARK scheme implemented in the Arkworks library [14]. We implement the key components of the proposed design and show their performance in practical scenarios through experiments. We highlight that the proposed data market design is modular and each of the proof-of-concept building blocks (contribution assessment, outlier detection, and the ZKP protocols) can be modified with different trade-offs.

Related works. Traditional data marketplaces were proposed for ML applications [15, 16, 17] considering ML model pricing and/or compensations for data owners. In [17], data owners are compensated based on the quality of their data along with their privacy sensitivity. Unlike our work, in [17], not only raw user datasets are observed by the data market but also all parties in the marketplace are assumed to be honest, i.e., there is no verification mechanism on the operation of the data market. Further, in such conventional designs, during data transactions, data market becomes a fully trusted and authoritative third-party [18], which may be undesired in many practical collaborative ML scenarios as it constitutes a single point of failure. To resolve this, references [18, 19, 20, 21] focus on building decentralized data marketplaces but fail to address potential data/model privacy issues. Focusing on privacy, in [22, 23, 24] blockchain-based data market schemes are implemented such that encrypted data are uploaded on the blockchain network. The problem here is two-fold. First, there is a memory limitation at each block so that the blockchain cannot support large ML models. Second, when the uploaded data are encoded, the system can only compute simple operations (such as linear combination of data) and cannot generalize to more complex and practical ML computations. FL in decentralized frameworks such as blockchain is studied in [25, 26, 27, 28, 29, 30]. Reference [25] discusses a serverless blockchained FL system, called BlockFL, where the role of the server in FL (i.e., gathering local updates and model aggregation) is performed by the blockchain network. Their proposed approach rewards the participants based on their dataset size. In [26], a blockchain-based FL system is proposed that uses differential privacy to protect data privacy. The reward mechanism in [26] is distance-based which may not be suitable for non-i.i.d. data distributions among the data owners. A contribution-based FL system on blockchain is proposed in [27] by only considering i.i.d. data distribution. Many existing works in collaborative/federated learning [17, 28, 29] use Shapley value (SV) associated with each party to evaluate the contributions of the participants. In [28], authors propose a group-based SV approach that is suitable with secure aggregation techniques for contribution evaluation. In [29], authors design a blockchain-based peer-to-peer payment system called FedCoin that rewards participants based on their SVs. We want to highlight that in practical collaborative ML systems, computing the SV is not feasible when the number of data owners is large. Recently, [30] proposed a blockchain-based data market named OmniLytics that gives the same reward to all honest data owners, which (albeit being efficient) has fairness implications and is prone to alterations by adversarial participants.

Limitations of the existing data market designs. Existing data market designs for collaborative/federated learning suffer to simultaneously provide: i) contribution-based fair rewards, ii) privacy, iii) robustness against malicious parties, iv) verifiability, v) efficiency and universal applicability to heterogeneous data distributions and different learning tasks, all of which motivate us to propose a novel blockchain-based data marketplace for collaborative ML.

II Problem Setting and Contributions

Goals. Our goal is to design a blockchain-empowered data marketplace in which i) the data owners get rewarded fairly based on the contribution of their locally trained models, ii) any data movement from the data owners is not allowed and no other party can see the local datasets of the data owners, iii) robustness is provided against adversarial behavior of the malicious participants, iv) all of the computations including model aggregation, contribution assessment, and outlier detection are verifiable, and v) the implementation is efficient and applicable to practical scenarios for different learning tasks under varying degrees of data heterogeneity.

Proposed Data Market Overview and Contributions. We design a trustworthy proof-of-contribution-based data marketplace for collaborative ML on blockchain that satisfy all these goals mentioned above (see Fig. 2). The proposed system consists of a project owner, data owners (trainers), an aggregator, and a distributed storage infrastructure. We use a decentralized P2P storage infrastructure called the Interplanetary File System (IPFS) as in [31]. For each stored file, IPFS generates a unique content identifier (CID), which points at the content itself (as opposed to its location in centralized web) that is used to access the file.

The project (model) owner would like to crowdsource the training of its model to interested data owners, i.e., trainers, in exchange for monetary returns. The data market commences with the project owner creating a smart contract by depositing some tokens and task descriptions, i.e., CIDs of validation/test datasets (which are already uploaded to IPFS), model description, training hyperparameters to initialize the project. With the execution of the smart contract on blockchain, interested data owners join the system. Once sufficiently many trainers join the system, training starts by the aggregator initializing the model and storing the base model in IPFS. Each participating trainer pulls the current global model from the storage, performs local training, and uploads its local model to IPFS. The aggregator retrieves the corresponding local models and performs outlier detection/removal and weighted aggregation of the accepted honest local models as well as an evaluation of the aggregated model on the validation dataset (for hyperparameter tuning). The aggregator then submits the benign data owner IDs (addresses) and zero-knowledge proofs (ZKPs) for outlier detection and the model aggregation results to the smart contract, which are then verified by the smart contract. If the verification is successful, the aggregator is notified to start the next round of training. Otherwise, the training run is terminated. The system continues for a certain number of rounds or until the aggregator determines that the performance is saturated.

Once the training is completed, the aggregator performs contribution assessment of the trainers222Contribution assessment is performed by the aggregator at the end of training using the best training run. and generates ZKPs for the contribution assessment as well as the per round validation accuracy computations performed during training. These proofs and the contribution vector are submitted to the smart contract, which then performs the verification. If the contribution assessment and per round validation accuracies are verified, the rewards are distributed from the project owner to the participating trainers based on the contribution of each trainer. In distributing the rewards, we give a fixed fee per round to each participating honest trainer in addition to the contribution-based payment. In our design, the exact amount of total reward (to be distributed from the tokens deposited by the project owner) is based on the accuracy of the final model.

The key features of the proposed data market are as follows:

Contribution-based reward allocation mechanism: The total reward is based on the accuracy of the final model whereas the individual rewards of each trainer is based on how much each of them contributes to the overall performance of the final model. Thus, not only the project owner has control over the total reward payment but also trainers are compensated fairly. 2. 2.

Privacy: In our marketplace, no party learns about the private datasets of the data owners since datasets are not required to be moved from the data owner (trainer) side. Since the local models stored at IPFS are encrypted, only the aggregator can see the individual models of the trainers. Further, trainers may opt for differential privacy [32] in uploading their local models to the storage. 3. 3.

Robustness: The use of blockchain guarantees that the project owner cannot evade payment after training. Thanks to the employed verification mechanisms, at each step of the data market execution, the proposed design is robust against malicious parties including but not limited to i) malicious trainers poisoning their models to disrupt the convergence of the global model or make the final result incorrect, ii) a malicious aggregator uploading incorrect aggregation, outlier detection, and/or contribution assessment results, and iii) a malicious project owner that wants to evade payment. 4. 4.

Verifiability: The integrity of the model aggregation, contribution assessment, and outlier detection computations are verified via ZKPs generated by the aggregator. 5. 5.

Efficiency and universal applicability: The proposed scheme can be implemented with the existing tools such as ZKPs for matrix multiplication and blockchain. Unlike most of the prior blockchain-based works (e.g.,[28, 30, 26, 27]), in our design, computations are not repeated by the miners for verification purposes. In our design, just the aggregator performs the computations off-chain and the smart contract verifies the results without doing heavy computations on-chain, which in turn makes our design lightweight and efficient in implementation. In addition, the proposed data market has a modular design and is applicable to practical scenarios with data heterogeneity and trainer dropouts. Our design can be utilized for various learning tasks as we do not implement any encryption on the trainer data.

III Data Market Design: Building Blocks

III-A *Proposed Contribution Assessment Metric *

To have a fair resource/credit/reward allocation [29, 33] in collaborative ML, one needs to assess the contribution of each local model. There are various methods for computing the contribution of the locally trained models in the literature [33, 34, 35]. In [34], Shapley value (SV) [36] associated with each local model is used as a contribution assessment metric. The main idea of SV is to compute the marginal utility of each model (i.e., each trainer) considering different trainer permutations. In collaborative ML, this utility function is the test accuracy. The main disadvantage of SV is that one needs to retrain the model for each permutation to compute the exact SV, which incurs a significant computation load (that is exponential in the number of federated trainers), thus making exact SV impractical in collaborative ML as a metric for contribution assessment.

Efficient approximations of the SV method has been studied in the literature [33, 37, 38]. Among these, the current state-of-the-art SV approximation technique is GTG-Shapley [38]. In GTG-Shapley, local model updates are used to reconstruct the corresponding model under different trainer permutations (as opposed to retraining from scratch) and certain within round and between round truncation techniques are implemented to speed up the computation.

Randomized leave-one-out (RLOO). In our data market design, as mentioned earlier, the aggregator performs the contribution evaluation of the trainers. Considering the verifiability of this contribution evaluation, the smart contract needs to receive proofs for each accuracy evaluation performed by the aggregator (as detailed in Section III-C), in which case the number of permutations used for SV approximation burdens the system performance. Motivated by this, in our data market design, we propose an even more efficient randomized leave-one-out (RLOO) method (compared to GTG-Shapley) to assess the contribution of each local model. Inspired by the data point deletion method of [35], in RLOO, by removing one model from the aggregate at a time, we measure how the accuracy of the aggregate model changes to deduce the affect of a particular local model. In RLOO, instead of using the aggregate of all local models, the aggregator first samples a subset of the trainers and uses the aggregate of the local models of the sampled subset, hence the name randomized leave-one-out.

Let $N^{(t)}$ denote the number of honest trainers in training round $t$. To compute the contribution of trainer $i$ in round $t$, denoted by $C^{(t)}_{i}$, the aggregator first samples a $\log N^{(t)}$ subset of trainers (making sure that trainer $i$ is in the subset) and computes their aggregate model $\mathbf{w}^{(t)}$. Then

[TABLE]

where, $\mathbf{w}^{(t)}_{-i}$ denotes the aggregate model of the subset except the local model of trainer $i$. In (2), $\text{Acc}(\mathbf{x})$ is the accuracy function of model $\mathbf{x}$ over the test data. If (2) is a positive number, we say the $i$th data owner contributes to the model. When (2) is negative, this indicates that the global model is better off without the local model of the $i$th data owner in that training round. Finally, we normalize the contribution values $C^{(t)}_{i}$ of each honest data owner $i$ in each round $t$ and report the sums over the entire training run as total individual contribution values $C_{i}$ of the honest data owners.

III-B Proposed Outlier Detection Mechanism

Malicious trainers or outside attackers in collaborative ML can harm training by employing Byzantine attacks [39, 40] or backdoor attacks [41, 42]. Byzantine attacks aim to disrupt the convergence of the global model, while backdoor attacks aim to mis-classify some data to a wrong class via backdoors. Existing literature on Byzantine-resilient learning and mitigation of adversarial behavior includes [12, 43, 40, 44]. These existing approaches either require knowing the number of malicious trainers in advance [12] or re-weighting trainer submissions [44], which are not practical in real systems, since an adversary will not notify the system before attacking, and re-weighting will tamper the aggregation results if there is no attack. A successful outlier detection approach should simultaneously satisfy the following: 1) the approach should detect whether an attack has happened; 2) if that is the case, the approach needs to handle the poisoned models to mitigate their effects; and 3) the approach should not harm benign trainer submissions when handling outliers.

In this work, we propose a novel two-stage outlier detection mechanism satisfying all three properties. The proposed approach first performs a cross-round check to detect whether any attacks happened. In case of attacks, it then performs a cross-trainer check to remove malicious trainer submissions.

Cross-round check. The aggregator computes the cosine similarity between the local models submitted by each trainer in the current round and the previous round. Specifically, for trainer $i$ and the local model $\mathbf{w}_{i}^{(t)}$ in round $t$, it computes

[TABLE]

Any cosine score $S_{c}$ that is lower than a threshold $\gamma$ indicates that an attack happened in the current round.

**Cross-trainer check. ** In case of attacks, the aggregator performs a cross-trainer check to remove malicious models. The cross-trainer check inherits the ideas of the three sigma rule [11] and the Krum algorithm [12]. The three sigma rule indicates that in normal distributions the percentage of values within one, two, and three standard deviations of the mean are 68%, 95%, and 99.7%, respectively. Enabled by the central limit theorem [45] and certain transformations [46, 47, 48, 49], the three sigma rule has many real-world applications and has previously been used in outlier detection [50].

The main idea of the cross-trainer check is to compute the distance between each trainer model and the average trainer model. Using these distances, the aggregator computes a boundary for outlier detection and removes trainer models which lie outside this boundary.

In particular, the aggregator computes Euclidean distances $\lVert\mathbf{w}_{i}-\mathbf{w}_{\mathit{avg}}\rVert$, where $\mathbf{w}_{i}$ denotes the $i$th trainer model, $i\in[L]$, and $\mathbf{w}_{\mathit{avg}}$ is the average trainer model, computed as a “middle point”. In the first round of training, $\mathbf{w}_{\mathit{avg}}$ is computed using the m-Krum algorithm [12] by taking $m=\frac{L}{2}$, when the aggregator receives $L$ locally trained models, assuming we have honest majority. In other training rounds, the average is computed by simply averaging all benign trainer models.

We let $\mathcal{S}_{E}$ denote the list of the Euclidean distance scores with cardinality $|\mathcal{S}_{E}|$. The aggregator then computes an approximate normal distribution $N(\mu,\sigma)$ using $\mathcal{S}_{E}$ with

[TABLE]

Invoking the three sigma rule, the aggregator computes the outlier boundary as $\mu+\sigma$ so that only the trainer models with scores less than $\mu+\sigma$ are kept, i.e., the ones that are at most one standard deviation away from the mean. The others are deemed as outliers and removed (see Algorithm 1).

III-C ZKPs for Aggregator Computations

Introduced in [51], zero-knowledge proofs (ZKPs) enable a prover to convince a verifier that a computation on the prover’s secret input (called the witness) is correct via a short proof, without revealing any information about the secret input. ZKPs guarantee that the prover cannot cheat (integrity property) as well as that the prover’s secret input remains confidential (zero-knowledge property). Due to their strong integrity and privacy guarantees, ZKPs have been utilized in blockchain systems [52], and machine learning applications [53, 54, 55].

In the proposed data market design, heavy computations such as outlier detection, model aggregation, and contribution assessment are performed off-chain by the aggregator. Since the aggregator is essentially an untrusted server, it may send inaccurate/incorrect results in order to save its computation time and resources or due to malicious behavior. Thus, the smart contract requires proof of the aggregator’s computations. In our work, we utilize zero-knowledge Succinct Non-interactive Arguments of Knowledge (zkSNARKs). zkSNARKs offer proofs that are short in length and easily verifiable without interactions between the prover and verifier, thereby are desirable for blockchain systems in which the verifier time/resources are critical. In the subsequent text, the prover is the aggregator and the verifier is the smart contract.

ZKP-compatible language. The first challenge of applying ZKP protocols is to express the computations in ZKP-compatible languages. ZKP protocols model the computation as an arithmetic circuit with modular addition and multiplication gates over a prime field. However, the computations in our data market design are on real numbers, such as the outlier detection and the machine learning computations. To address this issue, in our system, we build ZKP schemes using the Arkworks library [14], which compiles the description of an arithmetic circuit in a front-end language similar to Rust to the back-end of ZKP protocol automatically. On top of the library, we implement a class of fixed-point numbers and operations. We also develop optimized gadgets for common computations used in our data market design, including the ReLU function, maximum and square-root. Compared to naive implementations of these functions as circuits, the number of gates is reduced significantly.

ZKP for outlier detection. A zero-knowledge proof for matrix multiplication constitutes the basis of the verification schemes we use for the outlier detection and the model aggregation in the proposed data marketplace design. To verify a matrix multiplication $AB=C$, where $A,B,C$ are of size $n\times n$, instead of proving the computation step-by-step with $O(n^{3})$ multiplication gates, we apply the Freivalds’ algorithm [56] to test the correctness of the result with a random vector. The verifier generates a random vector $v$ of length $n$, and then checks $A(Bv)\stackrel{{\scriptstyle?}}{{=}}Cv$ inside the ZKP scheme. The size of the circuit is reduced to $O(n^{2})$ from $O(n^{3})$ in the naive solution. This scheme requires a commit-and-prove zkSNARK [57] and the random vector is generated after the prover commits to the result of the matrix multiplication.

We utilize the idea of verifying the computation again to design an efficient protocol for square-root, which is used in Algorithm 1. To verify that $x=\sqrt{y}$ is computed correctly, we ask the prover to provide the result $x$ and check in the ZKP that $x^{2}$ is close to $y$. Note that as the ZKP works over a prime field, it is not correct to check that x^{2}=y\mod $p$. Instead, we represent $x$ and $y$ as fixed-point numbers using our new class and check that $x^{2}\leq y$ and $(x+1)^{2}\geq y$, where the “+1” denotes the adding one to the least significant bit of the fixed-point number represented in the prime field as an integer. In this way, the cost of a square-root computation is improved to two multiplications and two comparisons.

ZKP for model aggregation. The model aggregation takes the average of all the local models, which requires a division by the number of accepted (benign) trainers. To verify a fixed-point division $c=a/b$, we ask the prover to provide the quotient $c$ and the remainder $r$, and check that the $a=b\cdot c+r$ and that the remainder $r$ is less than the divisor $b$.

In model aggregation, we further observe that if all trainers behave honestly, the divisor is the number of trainers, a publicly known value. Thus, we ask all the trainers to divide their local models by the number of trainers before submitting them to the aggregator, which saves the cost of division completely. The trainers will tend to do so honestly since otherwise they can be detected as outliers or have a lower reward.

ZKPs for contribution assessment. In order to verify the contribution assessments performed by the aggregator, we utilize ZKPs for the accuracy evaluation of an ML model. Here, the goal is to prove the accuracy of a secret model on a committed dataset. We utilize the ZKP for matrix multiplications described above for the linear layer(s) of a logistic regression and a neural network to improve the efficiency significantly. In addition, we avoid computing the costly sigmoid and softmax functions in the last layer and instead select the maximum value from the vector directly. This does not change the result of the inference and the accuracy of the ML model.

Choice of the ZKP system. In our implementation, we use the Groth16 [13] zkSNARK scheme implemented in the Arkworks library [14] for all the computations described above. The main reason is that the scheme has the smallest proof size (128 bytes) and can be verified directly in Ethereum’s smart contracts with native instructions. There are recent papers on more efficient ZKPs for ML inferences and accuracy [55], but the proof there is too big to be posted on-chain and verified by a smart contract (hundreds of KBs). Achieving both efficient prover time and short proof size is left as an interesting future work. Our ZKP implementation is a proof-of-concept building block for the entire design of the data market, and we can switch to other existing ZKP schemes to implement the same computations with different trade-offs.

IV Data Market Design: Detailed Operation

In this section, we present the proposed blockchain-based data market for collaborative ML in more detail. We have a single project (model) owner, data owners (trainers), and an aggregator. The project owner initiates the data market protocol by deploying a smart contract on the blockchain. Then interested data owners join the system incentivized by the monetary returns offered in the contract by the project owner. The aggregator is a processing node that performs certain computations including assessing trainer contributions, removing outliers, updating hyperparameters, and so on.

In conventional schemes such as OmniLytics [30], the project owner uploads all the data as well as the initial model to blockchain. However, due to memory limitations, it is not possible to put all these on the blockchain network. That is why, in our design, we utilize a decentralized storage structure such as IPFS. Once a piece of information is stored on IPFS, a unique content identifier (CID), which is essentially a hash value, is provided by the storage. That CID is later on used to access that particular information. By utilizing IPFS, the project owner (and any other entity of the proposed data market for that matter) avoids storing the model, datasets or any other large data on the smart contract (on-chain). Instead, we only store the CIDs on the smart contract, overcoming any memory-related issues on blockchain. This is one of the main differences between the proposed data market and the conventional data marketplace schemes.

The execution of the proposed data marketplace is as follows (see Fig. 3 for a flowchart of our design).

IV-A Initialization

Start. In order to start a data market session, the project owner uploads its validation and test datasets to IPFS and deposits some tokens as well as creates a smart contract that includes the task descriptions for the learning task. These descriptions include the CIDs of the validation and test datasets, learning model description, training hyperparameters, and so on. The deployed smart contract specifies the total amount of reward that will be available to the trainers based on the performance of the final trained model on the test data.

Registration. With the execution of the smart contract on blockchain, the contract waits until $L$ data owners register for training the project owner’s model on their local data. Each participating data owner provides two important arguments: a public key that is used to perform asymmetric encryption for secure IPFS storage and P2P communication along with a blockchain account address to be used to distribute rewards (tokens) based on proof-of-contribution after training. Once $L$ data owners join, the contract moves to the training phase.

Remark 1:

In this work, our focus is on the trustworthy, verifiable operation and contribution-based rewards to the participating trainers. A separate open problem is selecting the trainers for the learning task at hand. For this, we envision that the smart contract can check the validity, i.e., whether that trainer has enough number of samples and good computational resources, and legality, i.e., good reputation, of each interested trainer. Trainers that do not meet the criteria in the smart contract would be rejected. How to select the best trainers in a verifiable/privacy-preserving manner in collaborative ML is beyond the scope of this work.

IV-B Training Phase - Common Operations Per Round

Training starts by the aggregator initializing the model weights and storing the base model in IPFS. The aggregator records the CID of the base model to the smart contract.

Local training. Registered trainers pull the base model from the storage using its CID and start local training per the project owner’s specifications. Once the local training finishes, each trainer encrypts and uploads its local model to IPFS and records the corresponding CID to the smart contract. Thanks to this encryption, only the aggregator who knows the shared secret can decrypt the local model of each trainer. At this stage, trainers can further protect the privacy of their models by adding DP noise [32]. There is a trade-off between obtaining rewards and privacy, i.e., if a trainer adds more noise, then the performance of the local model reduces, thereby possibly decreasing that trainer’s monetary return from training.

Outlier detection. When the locally trained models are uploaded by the trainers, the aggregator downloads the local models and decrypts each of them by using a symmetric Diffie-Helmann (DH) secret derived from the trainer’s public key and the aggregator’s private key. Upon decrypting the local models, the aggregator performs outlier detection as described in Section III-B. Next, the aggregator generates a proof for its outlier detection computation as described in Section III-C. Once the outlier detection is performed, the aggregator records the benign trainer IDs (addresses) as well as the associated proof on the smart contract. The IDs of the benign trainers can later be utilized to adjust the reputation of the trainers.

Model aggregation. After removing outliers, the aggregator performs weighted aggregation of the honest models. Once the aggregated model is computed, the aggregator evaluates it on the validation dataset to guide the trainers to modify/tune the training hyperparameters. If the accuracy is not as desired, the aggregator can terminate that particular training run early. Upon aggregation, the aggregator writes the updated model to IPFS and uploads the corresponding CID, possibly modified hyperparameters, and a ZKP for the model aggregation (as described in Section III-C) to the smart contract.

Verification of outlier detection and model aggregation. The smart contract uses the corresponding ZKPs to verify the integrity of the outlier detection and the model aggregation results. If the verification is successful, the aggregator is notified to start the next training round. In the next round, each trainer uses the updated model and hyperparameters for local training, and the above steps are repeated for a certain number of rounds or until the aggregator determines that the performance is saturated.

IV-C Training Phase - Final Round

When the training is completed, the project owner releases the test dataset stored on IPFS to the aggregator so that the aggregator computes the performance of the final trained model. Using this, the smart contract determines the total contract reward. Here, the exact amount of the total reward to be distributed is based on the improvement in the project owner’s model and is specified in the smart contract.

Contribution assessment. The aggregator evaluates the contribution of data owners in each training round as in Section III-A. It performs this contribution evaluation using the test dataset at the end of the training (as opposed to at the end of each round) as it uses the best training run to compute contributions (in order not to perform any unnecessary heavy computations on the unused training runs that yield worse models). Once the contribution vector of the honest trainers $\mathbf{C}=[C_{1},C_{2},\dots,C_{L^{\prime}}]$ is determined, the aggregator submits it to the smart contract along with a proof as described in Section III-C. Here, $C_{i}$ denotes the $i$th honest trainer’s total contribution and $L^{\prime}$ out of the total $L$ trainers are honest.

In addition, since the aggregator has performed per round validation accuracy computations throughout training (for hyperparameter tuning), it sends an additional proof for these computations to the smart contract. This proof is also based on the ZKP for accuracy evaluation described in Section III-C.

Verification and reward spreading. If the contribution assessment and validation accuracy computations are verified, the rewards are distributed from the project owner to the participating honest trainers based on the contribution of each trainer. In addition, the project owner compensates the trainers based on their participation. That is, even if an honest trainer does not improve the model, that trainer is still rewarded for participation. The reward of honest trainer $i$ is then given by

[TABLE]

where $R_{p}$ is the fixed reward for participation, $\alpha$ is a parameter that depends on the total reward to be given (based on the accuracy improvement of the model), and $\bar{C}\triangleq\max(0,C)$. Here, we use $\bar{C}_{i}$ instead of $C_{i}$ since trainers with negative final contribution would not get any reward other than fixed $R_{p}$. Malicious trainers are not rewarded. As specified in the smart contract, based on the final accuracy of the trained model on test data, the project owner may get back some of the tokens he/she deposited in the beginning in case the model is not sufficiently improved by the trainers.

In addition to the participating honest trainers, the aggregator also receives monetary compensation for its operations during the execution of the marketplace. The aggregator is operated by an independent MLOps platform and paid separately by the project owner, according to an agreement between the project owner and the aggregator.

Remark 2:

In our design, validation and test datasets are provided by the project owner. These two datasets determine the final accuracy of the global model as well as the individual trainer contributions. A cheating project owner may purposefully send a biased or imbalanced test dataset (that is not similar to the validation dataset) to pay less to the trainers.333The project owner is not motivated to send an inaccurate validation dataset as this would hurt the training and yield a worse final model. To solve this issue, we have the aggregator randomly partition the project owner’s data. For this, in the beginning of the data market session, the project owner puts its data to IPFS and the corresponding CIDs are stored on the smart contract. Instead of letting the project owner specify the validation and test datasets, this time, we use a random seed generated by the trainers (through joint random number sharing algorithm [58] or random number generation algorithm on blockchain [59, 60]) in partitioning project owner’s data into test and validation datasets so that the validation and test datasets are of similar (if not the same) distributions.

Remark 3:

In our design, the smart contract used to execute the entire data market operation is deployed by the project owner in the beginning. In order to avoid a project owner deploying a malicious smart contract, we utilize a factory contract that project owners can use to instantiate contracts for their projects. As long as the factory contract functions as intended, smart contracts it initiates should behave as expected.

V Experiments

We provide experiment results to demonstrate the effectiveness/feasibility of the building blocks of the proposed design. In the outlier detection and contribution assessment experiments, we simulate the trainers and the aggregator on our in-house 64-bit Ubuntu 20.04.2 LTS machine equipped with AMD EPYC 7502 CPU.

V-A Contribution Assessment

To demonstrate the performance of the proposed RLOO metric, we use the same setup as in GTG-Shapley work [38] and train a multi-layer perceptron (MLP) model on MNIST dataset, that is comprised of handwritten digits. We consider 1) i.i.d. distribution where each trainer has the same distribution and the same number samples; 2) a highly non-i.i.d. scenario where each trainer still has the same number samples but the digits (labels) in each trainer are mutually exclusive, i.e., each label is in only one data owner; and 3) one of the trainers (Trainer 1 in our case) has rate data (labels 0 and 1) while all the other trainers have i.i.d. distributions from the remaining labels (labels 2-9). For all these cases, we plot $\bar{C}_{i}/(\sum_{j\in[L^{\prime}]}\bar{C}_{j})$ with $L^{\prime}=L=5$ in Figs. 4-6. Here, we want to highlight that unlike the GTG-Shapley approach [38], the proposed RLOO method is not meant to approximate the SV. We propose RLOO as an alternative more efficient metric for ZKP generation purposes. In the i.i.d. case in Fig. 4 we see that the GTG-Shapley method closely approximates the SV and the proposed RLOO yields consistent results where each trainer has similar contribution values as expected. On the other hand, in the non-i.i.d. case Fig. 5, the GTG-Shapley can no longer approximate the SV as closely and yields negative contribution values for trainers 3 and 5 (i.e., these trainers receive zero reward under GTG-Shapley) while the results produced by RLOO are more consistent. In Fig 6, we provide the results for a non-i.i.d. distribution where only Trainer 1 has labels 0 and 1. Unlike the GTG-Shapley scheme, RLOO and SV assign positive contributions (albeit less from the rest) to Trainer 1 which has rare data. GTG-Shapley scheme fails to acknowledge the contribution of Trainer 1. The key takeaway from these three experiments is that there is no universal way to accurately and effectively evaluate the trainer contributions in collaborative ML yet but the proposed data market in this work is compatible with various existing approaches to trainer valuation in collaborative ML.

V-B Outlier Detection

We set the cosine similarity threshold $\gamma$ to 0.3 in the cross-round check. That is, if the cosine similarity score in (3) is lower than 0.3, the aggregator deduces that there is an attack in the system. We consider two attacks: 1) Byzantine attack that produces random models for malicious trainers and 2) backdoor attack that scales up the weights of a malicious model to dominate the global model [41]. In outlier detection experiments, we train a logistic regression (LR) model on the MNIST dataset and a more complex ResNet-56 model on the CIFAR10 dataset for both attack types using $8$ trainers for a total of 10 rounds. We set the number of malicious trainers to $3$ in the Byzantine attack scenario and $1$ in the backdoor attack case. We present the accuracy of our approach to successfully detect attacks (cross-round accuracy), as well as the accuracy of correctly removing the poisoned models (cross-trainer accuracy) in Table I. Here, we would like to highlight that the accuracy values listed in Table I are not test accuracies of the model but the accuracies of detecting and removing outliers, respectively. A 100% accuracy means that attacks are detected and all outliers are removed. The experiment results provided in Table I demonstrate the effectiveness and potential of the proposed outlier detection approach in collaborative ML.

V-C ZKP Performance

We implement the ZKP system in Rust using the Arkworks framework [14]. The system consists of a machine learning module where various layers such as linear, embedding, and ReLU are implemented, and also an integration module that contains the outlier detection and the contribution assessment blocks. We test our system on Amazon AWS, using an m5a.4xlarge instance with 16 CPU cores and 32 GB memory.

We first test the inference of different models. In our experiments, we include LR and MLP models along with the Deep Learning Recommendation Model (DLRM). We sample a batch of 100 data points from the MNIST dataset and the Criteo Terabyte Dataset as our test set. We report the performance of a single model inference in Table II. We note that the proving time ranges from 4 seconds to 400 seconds depending on the complexity of the model, but the verification time remains the same thanks to the Groth16 [13] proving system, which helps the smart contract deployment.

After that, we integrate the model into the entire system with outlier detection, model aggregation, and contribution assessment using RLOO. We report the performance in Table III. For the collaborative training of an LR model, we manage to prove its correctness in fewer than 1.5 minutes, while for the most complex model, i.e., DLRM, proving will take roughly 2.5 hours. Verification time still remains constant.

While the cost of each part differs depending on the data and model, we note that in general, the bottleneck of the system comes from the contribution assessment part, i.e., RLOO. It requires multiple times of model aggregation and model inference, which are extremely costly in practice. We show the percentage of circuit size in Fig. 7. Optimizing the circuit size for ML models is an open problem in the ZKP research community, and we expect further improvement with the progress of the research.

VI Conclusion

We proposed a trustworthy proof-of-contribution-based data marketplace design for collaborative/federated learning on blockchain. Key features of our design include contribution-based fair reward allocation, data privacy, robustness to malicious activities, verifiability through ZKPs, and universal applicability. We proposed novel outlier detection and proof-of-contribution techniques as well as a proof-of-concept ZKP implementation as building blocks. Our data market design is modular in that we can switch to other existing schemes in these building blocks and implement the data market with different trade-offs. Current focus is on the end-to-end deployment of the proposed design.