Motivating Workers in Federated Learning: A Stackelberg Game Perspective

TL;DR

This paper proposes an incentive mechanism modeled as a Stackelberg game to motivate heterogeneous workers in federated learning, improving convergence by balancing worker diversity and training latency.

Contribution

It introduces an analytical equilibrium solution for incentivizing workers in federated learning using a Stackelberg game framework.

Findings

Optimal number of workers depends on budget and trade-offs.

Limited budget requires careful worker selection to balance diversity and latency.

Analytical solution guides incentive design for federated learning.

Abstract

Due to the large size of the training data, distributed learning approaches such as federated learning have gained attention recently. However, the convergence rate of distributed learning suffers from heterogeneous worker performance. In this paper, we consider an incentive mechanism for workers to mitigate the delays in completion of each batch. We analytically obtained equilibrium solution of a Stackelberg game. Our numerical results indicate that with a limited budget, the model owner should judiciously decide on the number of workers due to trade off between the diversity provided by the number of workers and the latency of completing the training.

Equations

$$\Delta (q_i,P_i)=V\mathbb{E}\left[\underset{i}{\max }{T}_{i,t}\right]+\sum _{i=1}^K{q}_i{P}_i,$$

$$\mathbb{E}\left[\underset{i}{\max }{T}_{i,t}\right]=\sum _{S\subseteq \{1,2,\dots ,K\}}(-1{)}^{|S|-1}\frac{1}{{\sum }_{i\in S}{\lambda }_i},$$

$$U_i(P_i,q_i)=q_i{P}_i-\kappa c_i(P_i{)}^2.$$

$$\displaystyle\max_{P_{i}}\ \$$

$$P_i\le P_{\max },$$

$$\displaystyle\min_{\boldsymbol{q}}\ \$$

$$\sum _{i=1}^K{q}_i{P}_i\le B,$$

$$\frac{\partial }{\partial P_i}{U}_i(P_i,q_i)=\frac{\partial }{\partial P_i}\left[q_i{P}_i-\kappa c_i(P_i{)}^2\right]=q_i-2\kappa c_i{P}_i,\forall i$$

$$P_i^{\ast }(q_i)=\{\begin{array}{ll}& \frac{q_i}{2\kappa c_i}\text{mbox}if\frac{q_i}{2\kappa c_i}\le P_{\max }\\ & P_{\max }\text{mbox}if\frac{q_i}{2\kappa c_i}>P_{\max }\end{array}$$

$$L(\mathbf{q},\alpha )=V\mathbb{E}\left[\underset{i}{\max }{T}_{i,t}\right]+\sum _{i=1}^K\frac{q_i^2}{2\kappa c_i}+\alpha \left(\sum _{i=1}^K\frac{q_i^2}{2\kappa c_i}-B\right),$$

$$\frac{\partial L(\mathbf{q},\alpha )}{\partial q_i}$$

$$=V\frac{\partial \mathbb{E}\left[{\max }_i{T}_{i,t}\right]}{\partial {\lambda }_i}\frac{1}{2\kappa c_i^2}+2{\lambda }_i{c}_i+2\alpha {\lambda }_i{c}_i,\forall i.$$

$$\alpha =-V\frac{\partial \mathbb{E}\left[{\max }_i{T}_{i,t}\right]}{\partial {\lambda }_i}\frac{1}{4\kappa c_i{\lambda }_i}-1,\forall i.$$

$$\frac{\partial \mathbb{E}\left[{\max }_i{T}_{i,t}\right]}{\partial {\lambda }_i}\frac{1}{{\lambda }_i}$$

$$\sum _{S\subseteq \{1,2,\dots ,n\}\text{ and }i\in S}(-1{)}^{|S|}\frac{1}{{\left({\sum }_{k\in S}{\lambda }_k\right)}^2{\lambda }_i}$$

$$\sum _{S\subseteq \{1,2,\dots ,n\}\text{ and }i\in S,j\notin S}(-1{)}^{|S|}\frac{1}{{\left({\sum }_{k\in S}{\lambda }_k\right)}^2{\lambda }_i}$$

$$\frac{q_i^2}{2\kappa c_i}=\frac{B}{K}\Rightarrow q_i^{\ast }=\sqrt{\frac{2B\kappa c}{K}}.$$

Full text

I Introduction

The recent success of deep learning approaches for domains such as speech recognition and computer vision stems from many algorithmic improvements but also from the fact that the size of available training data has grown significantly over the years, together with the computing power. The current trend is to use a larger data set and to train deeper networks (higher number of layers) to improve the accuracy. However, the complexity and the memory requirements quickly become unmanageable within the resources of a single machine. An efficient way to deal with this colossal computing task within a reasonable training time is to adopt distributed computation, and to exploit computation and memory resources of multiple machines in parallel. In [1], the federated learning system was introduced to allow the mobile devices perform computation of model training locally on their training data according to the model released by the model owner. Such a design enables mobile users to collaboratively learn a shared prediction model while keeping all the training data private on the device.

Most of the popular distributed training algorithms include mini-batch versions of stochastic gradient descent (SGD). Unfortunately, bulk- synchronous implementations of stochastic optimization are often slow in practice due to the need to wait for the slowest machine in each synchronous batch, i.e., they suffer from the so called straggler effect. For example, experiments on Amazon EC2 instances show that some workers can be five times slower than the typical performance [2]. There have been several attempts in the literature to mitigate the straggler effect by adding redundancy to the distributed computing system via coding [3, 4] or via scheduling computation tasks [5, 6]. However, these works overlooked the inherent heterogeneity in the computing capacity of different workers. It is crucial to consider the implications of such heterogeneity on optimizing the task allocation to different workers, improving learning accuracy, minimizing latency, and/or minimizing energy consumption. In that sense, [7] considers the problem of adaptive task allocation with the aim to maximize the learning accuracy, while satisfying a delay constraint resulting from data distribution/aggregation over heterogeneous channels, and local computation on heterogeneous nodes. Furthermore, the limited computing resource of a user device is shared among all running applications. The independent and rational mobile clients need an incentive to participate in federated learning. Hence, a critical question that needs to be addressed by each worker is “How much Central Processor Unit (CPU) resource of heterogeneous workers should be allocated to the training task of the model owner?” The answer to this question has repercussions for the central model owner, since in its most basic form of synchronous SGD, the model owner has to wait for the gradients from all workers in order to update its current set of model parameters.

In [8], a game theoretical approach is established to consider a communication incentive in federated learning, where the aim was to construct a relay network and cooperative communications for supporting model update transfer. Unlike the previous works, in this paper we consider an incentive-based approach to motivate the workers to allocate more computation power for local training. In this setting, at each gradient update step, the model owner offers an incentive to each worker participating in the federated learning process. Based on this incentive, the workers determine the CPU power they will use to calculate their gradient from the local data. The model owner has a finite budget and distributes its budget among its workers to achieve a fast convergence to a target error rate. We model the interaction between the mobile devices and the model owner as a Stackelberg game. In Stackelberg game, the model owner is the buyer as it buys the learning service provided by the mobile devices. Then, the mobile devices that are the service providers act as the sellers. The model owner inherently acts as the single leader in the upper level of the Stackelberg game while the mobile devices are the corresponding followers. We obtained the equilibrium solution of this game by first quantifying the average time required to finish a single iteration of SGD. We also implemented our game theoretical algorithm numerically on MNIST dataset. Our analysis provides insights on the optimal number of workers to achieve a desired balance of error-latency tradeoff.

II System Model

We consider a cooperative federated learning system as shown in Figure 1. Specifically, a model owner employs a set of mobile devices, i.e., workers to train a high-quality centralized model. The workers fetch the current parameters $\boldsymbol{w}_{i}$ from the model owner as and when instructed in the algorithm. Then, they compute gradients using one mini-batch and push their gradients back to the model owner. At each iteration, the model owner aggregates the gradients computed by the workers and updates the parameter $\boldsymbol{w}$.

Let $T_{i,t}$ be the time elapsed for the worker $i$ to update the gradient in iteration $t$. Here, we consider plain synchronous SGD such that the model owner waits for all the workers to push their gradients. Thus, iteration $t$ is completed in $\max_{i}T_{i,t}$ time, when all workers send their gradient updates. We assume that the time taken by a worker $i$ to compute gradient of one mini-batch is random and independently distributed across mini-batches and workers [9]. Specifically, we assume that $T_{i,t}$ is exponentially distributed with mean $\frac{P_{i}}{c_{i}}$, where $c_{i}$ denotes the total number of CPU cycles required to accomplish the computation task, and $P_{i}$ denotes CPU power, i.e., the computation ability represented as CPU cycles per second of worker $i$.

The model owner negotiates with the workers about the CPU power, i.e., $P_{i}$. In return, each worker $i=1,\ldots,K$ will receive the revenue $q_{i}P_{i}$ from the model owner, where $q_{i}$ is the price of one unit of worker $i$’s CPU power. Intuitively, the latency required to finish the learning process depends on the total usage of CPU power of all workers. Specifically, the learning latency becomes smaller as the expected value of maximum of $T_{i,t}$ reduces. As a result, the model owner aims to minimize the following cost function:

[TABLE]

where $V\geq 0$ is a positive constant optimization parameter. Note that ${\mathbb{E}}\left[{\max_{i}T_{i,t}}\right]$ decreases with increasing value of $P_{i}$. Let $B$ denote the maximum allowable budget of the model owner to pay for CPU power usage of cooperative workers.

Lemma 1

The expected value of the time required to finish a single iteration is obtained as:

[TABLE]

where the outer sum is over all non-empty subsets $S$ of $\{1,2,\ldots,K\}$ and $|S|$ denotes the number of elements of $S$. In addition, $\lambda_{i}=\frac{P_{i}}{c_{i}}$.

*Proof: We omit the proof due to lack of space, but it follows the same lines of derivation of Proposition 3.2 in **[10*].

Note that as the workers obtain a revenue of $q_{i}P_{i}$ from the model owner, each model device has an energy cost incurred from the computation, which is directly dependent on the value of CPU power usage, $P_{i}$ as: $\kappa c_{i}(P_{i})^{2}$, where $\kappa$ is a coefficient depending on the chip architecture [11]. Thus, the objective of each worker $i$ is to maximize the following utility function:

[TABLE]

where $P_{\max}$ is the maximum of allowable CPU power usage of the workers.

III Stackelberg Game Formulation and Equilibrium Solution

Next, we model the interaction between the workers and the model owner as a Stackelberg game. In the lower level of the game, the workers determine their CPU power, $P_{i}$, as a function of price per unit, $q_{i}$. In the upper level, the model owner decides on the price per unit power for each worker, $q_{i}$. As a result, the Stackelberg game can be formally defined as follows:

1. Lower-level Subgame: Given the fixed vector of prices of one unit of CPU power $\boldsymbol{q}=q_{1},q_{2},\ldots,q_{K}$, the lower-level subgame problem is defined as:

[TABLE]

where $P_{\max}$ is the maximum available CPU power.

2. Upper-level Subgame: After each worker’s CPU power utilization with respect to prices, the model owner forms a upper-level subgame problem as defined as:

[TABLE]

where $B$ is the available budget to pay the workers.

Based on the game formulation, we consider a Stackelberg equilibrium to the solution for the model owner and the workers. Specifically, by following the backward induction, we firstly use the first-order optimality condition to obtain the optimal solution to the lower level subgame. Then, we substitute the Nash equilibrium of the lower-level subgame into the upper-level subgame and investigate the solution to the upper-level subgame.

III-A Solution to Lower-level Subgame

To find the optimal solution for the lower-level subgame at each worker, we take the first derivative of the utility function of each worker in (3) with respect to $P_{i}$:

[TABLE]

By equating (8) to zero, we obtain the optimal CPU power as:

[TABLE]

Furthermore, it is easy to show that the utility function of each worker is strictly concave, which guarantees the existence and uniqueness of nash equilibrium.

III-B Solution to Upper-level Subgame

After obtaining the optimal CPU power of each worker as a function of price per unit CPU power, we investigate the solution to the upper-level subgame for the model owner. Due to the high non-linearity of the maximum time equation given in Lemma 1, we cannot obtain the closed form solution for the general case. Instead, we present Lemma 2 that can be used to develop an efficient update algorithm to reach the equilibrium point for heterogeneous case, where $c_{i}\neq c_{j},\ \forall i,j\in\{1,2,\ldots,K\}$.

Lemma 2

When $V$ is sufficiently large, the optimal solution is realized at the boundary, i.e., $\sum_{i=1}^{K}\frac{(q_{i}^{*})^{2}}{2\kappa c_{i}}=B$, where $q_{i}^{*}$ is optimal budget allocation per unit of power for worker $i$.

**Proof: **The proof is given in Appendix A.

The closed-form solution to the homogeneous case where $c_{i}=c_{j},\ \forall i,j\in\{1,2,\ldots,K\}$ is given in the following theorem.

Theorem 1

*When $c_{i}=c_{j}=c,\ \forall i,j\in{1,2,\ldots,K}$, the optimal solution to the upper-level subgame is $q_{i}^{*}=\sqrt{\frac{2B\kappa c}{K}},$ for all $i$. ***Proof: **The proof is given in Appendix B.

IV Numerical Results

In the simulations, we use MNIST dataset for which we first convert the 28 x 28 images into single vectors of length 784. We use a single layer of neurons followed by soft-max cross entropy with logits loss function. Thus, effectively the parameters consist of a weight matrix $\boldsymbol{W}$ of size 784 x 10 and a bias vector $\boldsymbol{b}$ of size 1 x 10. We use a regularizer of value 0.01, and learning rate of value 0.05. For implementation we used Tensorflow with Python3. For the run-time simulations, we generate random variables from the respective distributions in python to represent the computation times. Specifically, the computation time for each worker is generated from an exponential distribution with mean $\frac{P_{i}}{c_{i}}$. Furthermore, to consider heterogeneous workers in the system, we select $c_{i}$ uniformly at random in the range of $[0.5\cdot 10^{3},1.5\cdot 10^{3}]$. Due to the randomness of the selection of $c_{i}$ and randomness of stochastic gradient descent (SGD), we run each realization 50 times and take the average. In the simulations, we are interested in the error rate defined as ratio of the difference of the processed image and the original image with the original image. In all simulations, we defined a target error rate, and if the target error rate is realized, we stop the simulations and consider the time elapsed to reach the pre-defined error rate.

We first investigate the effect of varying number of workers and budget on the latency in Fig. 2a. In all budget values, the latency initially decreases with the number of workers, since the error improves with increasing $K$, and thus, the number of workers in the training data set process leads to increase diversity and to reach the target error value, the system requires fewer iterations. However, after a certain point, the latency starts to increase. This is due to the fact that as the number of helpers increases, the positive effect of diversity of training data diminishes and the delay resulted in waiting for the update of all workers, starts to dominate. We also observe that the time required to reach the target error rate decreases, as the budget of the model owner increases. This is because an increase in the budget results in more CPU power allocated per worker reducing the time to complete each iteration. As a result, the total latency decreases. Fig. 2a demonstrates the trade-off between the diversity, which leads to reduction in total number of iterations, and the time elapsed to complete each iteration, both of which increases with the number of workers. Henceforth, for a given budget and target error rate, there exists an optimal number of workers that should be employed by the model owner.

Next, we investigate the optimal number of workers minimizing the total latency for varying budget and target error rates. As depicted in Fig. 2b, an increase in the budget leads to an increase in the optimal number of workers, since as the budget increases, more CPU power can be purchased from more workers. Furthermore, as target error rate decreases, the optimal number of workers increases. This is because as the target error rate decreases, the number of iterations to complete the process increases, which in turn allows the diversity of training data provided by different workers to become more effective.

V Conclusion

In this paper, we have presented a Stackelberg game model to analyze the CPU allocation strategies of multiple workers as well as the budget allocation of the model owner in a synchronous SGD run by the model owner. Specifically, we have investigated the impact of the available budget and target error rate on CPU power utilization of workers and the convergence time of the learning process. We observe that even though higher number of workers leads to higher diversity in the learning process, there is a maximum number of workers beyond which the delay due to waiting for SGD update dominates. This result demonstrates the importance of an efficient resource allocation algorithms in a practical learning system.

One important direction of extension of this work is to consider a dynamic game formulation that arises when the dynamic channel and worker CPU conditions are taken into account. Additionally, the interactions between the model owner and workers depend on the learning approach implemented, e.g., AdaGrad, ADAM, etc. Although similar trade-offs exist regardless of the method implemented, it would be insightful to study the optimal number of workers depending on the method used.

Appendix A Proof of Lemma 2

We first substitute the optimal CPU power allocations for the workers, i.e., $P_{i}^{*}$, given as in (9) into the cost minimization problem given in (6) - (7). As the constraint in (7) is linear, we adopt the Lagrangian method. The Lagrangian function for the optimization problem (6) - (7) is given as follows:

[TABLE]

where $L(\cdot)$ is Lagrangian function and $\alpha$ denotes Lagrangian multiplier.

To obtain the optimal solution, we take the first derivative of Lagrangian function with respect to $q_{i}$.

[TABLE]

In (11), we use the relation between $\lambda_{i}$ and $q_{i}$, i.e. $\lambda_{i}=\frac{q_{i}}{2\kappa c_{i}^{2}}$. Then, we equate the first derivative given in (11) to zero to derive the value of Lagrange multiplier, $\alpha$, at the optimal point as:

[TABLE]

Similarly, from the second and third Karush-Kuhn-Tucker (KKT) conditions we have $\alpha\left(\sum_{i=1}^{K}\frac{q_{i}^{2}}{2\kappa c_{i}}-B\right)=0$ and $\alpha\geq 0$. Thus, the first term in (12) should be positive. Intuitively, as the exponential parameter, i.e., the inverse of mean completion time, $\lambda_{i}$, increases, the maximum value of completion times should decrease. As a result, for a sufficiently large $V$, we can guarantee that $\alpha$ is positive. Thus, from complementary slackness condition of KKT, the solution exists at the boundary.

Appendix B Proof of Theorem 1

Using the result given in (11) and the fact that $c_{i}=c_{j}=c,\ \forall i,j$, we obtain the following relation:

[TABLE]

A trivial solution to (13) is achieved when $\lambda_{i}=\lambda_{j}=\lambda,\ \forall i,j\in{1,2,\ldots,K}$. This will also result in equal value of $q_{i}$’s. Combining this result and the result given in Lemma 2, we obtain:

[TABLE]