Coded Distributed Computing Under Combination Networks

TL;DR

This paper introduces a new method for efficient communication in distributed computing systems using coded techniques in combination networks.

Contribution

A novel transmission scheme for the shuffle phase using a coefficient matrix and Combined Placement Delivery Array (CPDA) is proposed.

Findings

The proposed scheme improves uplink and downlink communication loads in (H,L)-combination networks.

The method maintains robustness and efficiency in combined multi-AP networks.

The approach outperforms baseline schemes in communication performance.

Abstract

Coded distributed computing (CDC) is a powerful approach to reduce the communication overhead in distributed computing frameworks by utilizing coding techniques. In this paper, we focus on the CDC problem in (H,L)-combination networks, where H APs act as intermediate pivots and K=HL workers are connected to different subsets of L APs. Each worker processes a subset of the input file and computes intermediate values (IVs) locally, which are then exchanged via uplink and downlink transmissions through the AP station to ensure that all workers compute their assigned output functions. In this paper, we first novelly characterize the transmission scheme for the shuffle phase from the view point of the coefficient matrix and then obtain the scheme by using the Combined Placement Delivery Array (CPDA). Compared with the baseline scheme, our scheme significantly improves the uplink and downlink communication loads while maintaining the robustness and efficiency of the combined multi-AP network.

Figures

Full text

1. Introduction

The rapid growth of computationally intensive applications has driven considerable research into efficient distributed computing frameworks. MapReduce [1,2] and its evolution Apache Spark [3] have significant advantages in computational tasks dealing with massive datasets, where the amount of data can usually reach tens of terabytes. The MapReduce framework distributes computation tasks to multiple nodes, each storing a subset of the dataset. The computation process is decomposed into a set of “map” functions and “reduce” functions. Each map function processes a batch of data to generate intermediate values (IVs), which serve as inputs to the reduce functions. The overall process comprises three main phases: map, shuffle, and reduce phases. The nodes compute the map functions on their locally stored data, generate output IVs during the map phase, and exchange their computed IVs to ensure that each reduce function accesses to all necessary inputs during the map phase. In the reduce phase, all the nodes compute their assigned reduce functions using the gathered IVs.

Due to massive data exchange and limited communication bandwidth, the MapReduce-type system (MapReduce in short) often suffers from communication bottlenecks, i.e., the data shuffle phase consumes a significant portion of the overall task execution time. In order to reduce the communication overhead during the shuffling phase, Li et al. in [4] proposed a CDC scheme which achieves an optimal tradeoff between the computation load and communication load under a given output function assignment. The CDC scheme has also been widely studied in the other distributed computing scenarios. For example, CDC has been combined with maximum distance separable (MDS) codes to handle matrix–vector multiplication tasks and mitigate the impact of stragglers [5]. There are also many workers focusing on the stragglers in general functions [6,7], optimal resource allocation strategies [8,9], iterative computing and shuffling procedures [10,11], randomized file allocations [12], and cases with random network connectivity [13].

The CDC technique has been extended to wireless networks [14,15], where devices exchange information via wireless links. In such scenarios, owing to the decentralized nature of wireless networks, a central access point (AP) is typically required to facilitate data exchange, giving rise to uplink and downlink communication phases [16,17,18]. In real-world scenarios, we usually consider multiple APs. For example, a large place like a university campus needs to deploy multiple APs to realize campus network coverage. Moreover, a multi-AP system is able to distribute the load to multiple repeaters, thus reducing the risk of a single point of failure and improving fault tolerance.

This paper considers the CDC problem in a wireless $[eqn]$ -combination network [19] which consists of H numbers of APs and $[eqn]$ computing workers. Each worker is connected to a distinct subset of L numbers of APs through wireless links. The APs act as intermediate hubs, facilitating the exchange of IVs among workers over wireless channels. Each worker is assigned a subset of the input files, processes them locally to compute IVs, and then transmits some IVs to its connected APs. The APs aggregate and broadcast the received information to other workers, ensuring that all workers obtain the necessary IVs to compute their respective output functions. This multi-AP structure provides enhanced reliability and robustness in the face of AP or node failures. We are interested in the tradeoff between the communication load and computation load (defined as the average number of nodes processing each input file), and aim to improve the overall efficiency of the distributed computing system by exploiting computation resources at workers and creating additional opportunities for parallelism and multicast transmission.

In this paper, we first obtain a baseline scheme by using CPDA. Then, by novelly characterizing the scheme from the view point of linear algebra, the problem of designing a CDC scheme for combination network is equivalent to constructing a special matrix. Finally, using CPDA, we obtain an improved CDC scheme which has a lower communication load than that of the baseline scheme, for which the computation load does not increase.

Notations: We denote the set of positive integers by $[eqn]$ . For $[eqn]$ , the set $[eqn]$ is denoted as $[eqn]$ , and $[eqn]$ is used to represent the cardinality of a set or the length of a vector. Denote $[eqn]$ , i.e., $[eqn]$ is the collection of subsets of $[eqn]$ of size L.

2. System Model

We consider a $[eqn]$ distributed coded computing system for a combination network, where there are N input files $[eqn]$ , each of which has B bits, H APs $[eqn]$ without storage, K workers, each of which can store at most M files, and K output functions $[eqn]$ from N files, each of which is arranged to one worker. Each worker is connected to a unique L-subset of APs over the wireless channel. So $[eqn]$ and the user set can be written as $[eqn]$ . Each worker is connected to the other workers via these wireless channels through common APs. For instance, Figure 1 shows a combination network with $[eqn]$ and $[eqn]$ . For each $[eqn]$ , the output function $[eqn]$ maps the N input files into a streams of U bits, i.e., we have $[eqn]$ . The map function is defined as $[eqn]$ . For each $[eqn]$ , the map function maps the input file $[eqn]$ into intermediate value (IV) $[eqn]$ of T bits. Similarly, for every $[eqn]$ , we define the reduce function as $[eqn]$ , which maps the N IVs into a stream of U bits. Consequently, the output function $[eqn]$ is given by

$$[eqn]$$

as described in [4,18,20]. Thus, a $[eqn]$ distributed coded computing scheme for a combination network consists of the following three phases:

•Map Phase: Each worker $[eqn]$ stores M files, denoted by $[eqn]$ . We assume that each file is stored the same number of times. Based on its local stored files, worker $[eqn]$ can compute all the IVs $[eqn]$ .

•Shuffle Phase: Each worker $[eqn]$ is assigned to compute an output function $[eqn]$ , where $[eqn]$ . In order to obtain all the IVs $[eqn]$ , the worker needs to request its required IVs from the other workers through their common APs. So, the communication process consists of two phases: uplink and downlink steps. Before the transmission begins, each worker encodes the IVs $[eqn]$ as $[eqn]$ using a randomized Gaussian encoding scheme. We assume that the transmission strategy is one-shot linear, i.e., each transmitted coded IV in a transmission could be decoded by any required worker and its local IVs. At each time slot s, in the uplink step, we denote the signal sent from the worker $[eqn]$ by $[eqn]$ which satisfies the power constraint $[eqn]$ . Here, the worker can be silent, i.e., not send any data. Then, each AP $[eqn]$ receives the following signal:

$$[eqn]$$

where $[eqn]$ and $[eqn]$ denote the channel coefficients from worker k to AP h and the additive white Gaussian noise, respectively. We assume that $[eqn]$ is chosen from $[eqn]$ if worker k is connected to AP h; otherwise, $[eqn]$ . Additionally, $[eqn]$ follows a normal distribution $[eqn]$ if worker k is connected to AP h; otherwise, $[eqn]$ .

In the downlink step, each AP $[eqn]$ sends the received coded signal $[eqn]$ from its connected workers. The goal of the communication is to obtain all the IVs $[eqn]$ required by each worker $[eqn]$ , who is arranged to compute the output function $[eqn]$ .

Specifically, each worker $[eqn]$ receives the following signal from its connected APs

$$[eqn]$$

where $[eqn]$ is also the additive white Gaussian noise, and if h is connected to k, then $[eqn]$ follows a normal distribution with $[eqn]$ ; otherwise, $[eqn]$ . We can omit the additive noise terms $[eqn]$ and $[eqn]$ . This assumption is valid under the condition that the transmit power is sufficiently large to ensure a high signal-to-noise ratio (SNR).

•Reduce Phase: After receiving all the signals sent down by the AP, each worker $[eqn]$ can decode all its required IVs and compute the arranged output function $[eqn]$ .

Definition 1.  The following two types of loads are used as criteria for evaluating the merits of a $[eqn]$ distributed coded computing scheme for combination network:

• Computation load is defined as the total number of mapped files across the K workers, normalized by N, i.e., $[eqn]$ ;
• Communication load consists of the uplink load $[eqn]$ , where $[eqn]$ is the length of the message received by the AP h in the uplink phase, i.e., the total normalized number of bits received by the H APs in the uplink process, and the downlink load $[eqn]$ where $[eqn]$ is the length of the message sent by AP h in the downlink phase, i.e., the total normalized number of bits sent by the H APs in the downlink process.

The authors in [21] introduced the CPDA to realize a coded caching scheme for a combination network. In this paper, we will use it to construct our schemes.

Definition 2 (CPDA). For any positive integers H,* N*,* Z and L with $[eqn]$ and $[eqn]$ , let $[eqn]$ , an $[eqn]$ array $[eqn]$ over $[eqn]$ is called a $[eqn]$ CPDA if it satisfies the following conditions:*

• C1. The symbol "∗" appears Z times in each column;

• C2. Each $[eqn]$ occurs at least once in the array;

• C3. For any two distinct entries $[eqn]$ and $[eqn]$ satisfying $[eqn]$ , then $[eqn]$ ;

• C4. For any $[eqn]$ , the labels of all columns containing symbol s have a nonempty intersection.

• If each integer occurs exactly g times in $[eqn]$ , we call it g-regular CPDA.

Let us take the following example to further illustrate the concept of CPDA.

Example 1.  When $[eqn]$ 4 and $[eqn]$ 2, let us consider the following array.

$$[eqn]$$

Clearly each column has exactly $[eqn]$ 2 stars, and each integer occurs at least once in $[eqn]$ . So the conditions C1–2 hold. Let us consider the conditions C3–4. When $[eqn]$ 1, we can obtain the following subarray:

$$[eqn]$$

Clearly, the entries

$$[eqn]$$

Therefore,* the above two symbols 1 satisfy condition C3. The column labels corresponding to each integer 1 are $[eqn]$ , $[eqn]$ , and $[eqn]$ . Clearly*,* $[eqn]$ , which satisfy condition C4. Similarly, we can obtain the subarrays which only contain integers 2*,* 3*,* and 4*,* respectively and show that the conditions C3–4 hold. So*,* $[eqn]$ is a $[eqn]$ CPDA.*

There are many constructions of CPDAs in [21,22,23,24]. Here, we list some results which we will use in this paper as follows:

Lemma 1 ([21]). For any positive integers H, L, and t satisfying $[eqn]$ and $[eqn]$ , there exists a $[eqn]$ , $[eqn]$ , $[eqn]$ CPDA.

Lemma 2 ([22]). For any positive integers $[eqn]$ satisfying $[eqn]$ , $[eqn]$ , $[eqn]$ , $[eqn]$ and $[eqn]$ , there exists a $[eqn]$ CPDA.

3. Main Results

In this section, the baseline scheme is introduced. Then, we characterize the CDC scheme from the view point of a coefficient matrix, thereby obtaining an optimized CDC scheme for the combination network. Finally, we use CPDA to derive the required coefficient matrix, which significantly simplifies the system design and enhances its performance. In addition, we demonstrate that the proposed scheme achieves a lower transmission load than the baseline scheme.

When each output function is computed by exactly one worker, the authors in [25] showed that the CDC problem is equivalent to the coded caching problem for device-to-device (D2D) networks. In addition, the authors in [26] proposed that a coded caching scheme for a shared link can be used to generate a coded caching scheme for a D2D network. Using the same method in [26], we can generate a CDC scheme for a combination network based on the CPDA and obtain the following result.

Theorem 1 (Baseline scheme). Given a g- $[eqn]$ CPDA $[eqn]$ with $[eqn]$ and $[eqn]$ , there exists a $[eqn]$ baseline coded distributed computing scheme for a $[eqn]$ -combination network,* with a computation load of $[eqn]$ and uplink and downlink communication loads of*

$$[eqn]$$

Proof. We use the star positions in $[eqn]$ to assign the files to each worker as follows. When $[eqn]$ , worker k stores file $[eqn]$ . So worker k stores the following files:

$$[eqn]$$

In the map phase, the set of IVs computed locally by worker k is

$$[eqn]$$

According to Definition 1, the computation load is $[eqn]$ .Recall that each worker $[eqn]$ is assigned to compute the output function $[eqn]$ . From (5), worker k contains the required IVs in $[eqn]$ and needs the other required IVs in $[eqn]$ , each of which should be transmitted by the other workers.Now, we will use the integers in $[eqn]$ to design the transmission strategy in the shuffle phase. For each integer $[eqn]$ , we assume that s occurs exactly g times in $[eqn]$ . That is,

$$[eqn]$$

Denote the set of workers associated with s as $[eqn]$ and their respective connected AP sets as

$$[eqn]$$

By the condition C4 of Definition 2, there exists at least one integer, say $[eqn]$ , satisfying

$$[eqn]$$

Recall that in the map phase, each worker can encode the IV $[eqn]$ into $[eqn]$ using a random Gaussian coding scheme, where $[eqn]$ has a size of t bits. To facilitate efficient transmission, each IV $[eqn]$ in $[eqn]$ is divided into $[eqn]$ equal-sized packets. That is,

$$[eqn]$$

where $[eqn]$ denotes the packet of $[eqn]$ intended for worker $[eqn]$ .In the shuffle phase, each worker $[eqn]$ transmits the following encoded information of length $[eqn]$ bits to the AP $[eqn]$

$$[eqn]$$

where $[eqn]$ represents the sub-slot within time slot s allocated to worker $[eqn]$ for transmitting its encoded data. Upon receiving the message $[eqn]$ where the coefficient $[eqn]$ is the channel coefficient between worker k and AP $[eqn]$ and is chosen from $[eqn]$ under the independent and identically distributed condition, the AP $[eqn]$ broadcasts it to all workers in $[eqn]$ . Consequently, each worker $[eqn]$ receives the message

$$[eqn]$$

In the reduce phase, each worker $[eqn]$ needs to compute the output function $[eqn]$ . By Condition C3 of Definition 2 and Equation (5), each worker $[eqn]$ already possesses all IVs in $[eqn]$ except $[eqn]$ . Therefore, each worker $[eqn]$ can decode its desired IV $[eqn]$ from $[eqn]$ . After time slot s, each worker $[eqn]$ obtains the IV $[eqn]$ . After all S time slots, each worker $[eqn]$ obtains all the missing IVs $[eqn]$ , which are required to compute the output function $[eqn]$ . With all the necessary IVs, each worker $[eqn]$ proceeds to compute $[eqn]$ and completes the reduce phase.Now, let us consider the communication load. By Definition 1, the uplink and downlink loads are

$$[eqn]$$

Then, the proof is completed. □

3.1. Main Idea

In this subsection, we characterize the CDC scheme under the combination network from a matrix perspective. Our task is to compute $[eqn]$ output functions, where each worker $[eqn]$ owns a portion of the input file denoted as $[eqn]$ .

Map Phase: For each mapped file $[eqn]$ , worker k computes the IV $[eqn]$ for all K output functions. Consequently, the set of IVs available at worker $[eqn]$ is given by

$$[eqn]$$

The computation load in the map phase is given by $[eqn]$

Shuffle Phase: The shuffle phase consists of two processes: (1) workers upload IVs to the AP via wireless channels, and (2) the AP broadcasts the received IVs to the connected workers. Before transmitting the information, each worker encodes the IV $[eqn]$ into $[eqn]$ using a random Gaussian coding scheme. During time slot s, a set of workers $[eqn]$ uploads IVs to AP $[eqn]$ . Assuming that the signal sent by worker $[eqn]$ at time slot s is denoted as $[eqn]$ , the signal received by AP $[eqn]$ can be expressed as

$$[eqn]$$

where $[eqn]$ is the channel coefficient matrix for time slot s, and $[eqn]$ is the transmit signal vector for time slot s. To design $[eqn]$ , we define $[eqn]$ as the encoding coefficients for worker $[eqn]$ . Then,

$$[eqn]$$

where $[eqn]$ is the coefficient matrix for time slot s, and $[eqn]$ is the vector of encoded IVs. Thus, the message received by AP $[eqn]$ is

$$[eqn]$$

By designing the coefficient matrix $[eqn]$ , we can control the IVs uploaded by workers to AP $[eqn]$ , completing the uplink process.

Next, we describe the downlink process. In time slot s, AP $[eqn]$ broadcasts the received message $[eqn]$ to its connected workers. Each worker $[eqn]$ then receives

$$[eqn]$$

Using the IVs it already possesses, worker $[eqn]$ can decode the required information from $[eqn]$ .

After S time slots, all K workers acquire the missing IVs required for the reduce function. Specifically, each worker $[eqn]$ obtains

$$[eqn]$$

By combining their own IVs $[eqn]$ , each worker now possesses all the necessary information to compute the output function.

Since the messages in both the uplink and downlink processes are $[eqn]$ , and assuming each IV has a size of t bits, the communication load per time slot is t bits. According to Definition 1, the total uplink and downlink communication loads are $[eqn]$

Reduce Phase: After the shuffle phase, each worker $[eqn]$ computes the output function $[eqn]$ using all the IVs they possess.

Theorem 2.  Given an $[eqn]$ CPDA with $[eqn]$ , there exists a $[eqn]$ coded distributed computing scheme under a combination network,* with a computation load of $[eqn]$ and its uplink and downlink communication loads given by*

$$[eqn]$$

Proof. We use a $[eqn]$ CPDA to design both the input file assignment and the coefficient matrix for the shuffle phase.Before the computation task begins, the input file assigned to each worker $[eqn]$ is

$$[eqn]$$

In the map phase, each worker maps the input file it owns into K IVs $[eqn]$ . Thus, the IVs at worker $[eqn]$ are

$$[eqn]$$

Since each worker is allocated Z input files, the computation load according to Definition 1 is $[eqn]$ Before the shuffle phase, each worker encodes the IV $[eqn]$ into $[eqn]$ using a random Gaussian coding scheme, where $[eqn]$ has a size of t bits. In the shuffle phase, we consider an integer $[eqn]$ in the CPDA $[eqn]$ , and suppose s occurs g times in $[eqn]$ . Let $[eqn]$ . The set of workers involved in time slot s is $[eqn]$ , and the APs connected to all workers in $[eqn]$ are denoted as $[eqn]$ . An arbitrary AP $[eqn]$ is selected to handle the uplink and downlink communication in time slot s.In time slot s, we design the message received by AP $[eqn]$ as

$$[eqn]$$

Let

$$[eqn]$$

From (13) and (18), we need to design the coefficient matrix $[eqn]$ such that

$$[eqn]$$

Since $[eqn]$ , by (17), worker $[eqn]$ does not possess the IV $[eqn]$ . Thus, $[eqn]$ , and (19) can be written as

$$[eqn]$$

By linear algebra, i.e., given the row vector $[eqn]$ , we can always obtain a non-zero column vector $[eqn]$ of $[eqn]$ such that the ith coordinate is zero. So, we can always obtain the coefficient matrix $[eqn]$ in (20).In the uplink phase, each worker $[eqn]$ transmits the encoded information $[eqn]$ to AP $[eqn]$ . In the downlink phase, AP $[eqn]$ broadcasts $[eqn]$ to all workers in $[eqn]$ . From Condition $[eqn]$ of Definition 2 and Equation (17), each worker $[eqn]$ already possesses all IVs in $[eqn]$ except $[eqn]$ . Thus, each worker $[eqn]$ can decode its desired IV $[eqn]$ from $[eqn]$ . After S time slots, all K workers obtain the missing IVs necessary for computing the reduce function.Since the size of each IV $[eqn]$ is t bits, the communication load per time slot is t bits. The total uplink and downlink communication loads are

$$[eqn]$$

In the reduce phase, each worker $[eqn]$ computes the output function $[eqn]$ . Since $[eqn]$ , worker k utilizes the IVs of all N files for the $[eqn]$ -th output function. After the shuffle phase, each worker $[eqn]$ is given the IVs needed to compute $[eqn]$ . Therefore, all workers can compute their assigned output functions. □

3.2. Performance Evaluation

In this subsection, let us consider the performance of the proposed schemes by applying the CPDA in Lemmas 1 and 2 to Theorems 1 and 2, respectively. The resulting schemes are summarized in Table 1 and Table 2 respectively. In the following, let us consider the computation load and communication load of these schemes.

Figure 2 illustrates the relationship between the computation load and the communication load for the schemes in Table 1 and Table 2. Let us consider the distributed computing system with $[eqn]$ computing nodes and $[eqn]$ output functions under the $[eqn]$ -combination network. As shown in Figure 2, when the same class of CPDA is used, the main scheme of Theorem 2 achieves a lower communication load than the base scheme of Theorem 1 for the same computation load. This improvement results from the main scheme’s more efficient utilization of the multicast gain, which reduces the communication load.

3.3. Example for Achievable Scheme

We consider a combinatorial network with $[eqn]$ , $[eqn]$ , where $[eqn]$ and $[eqn]$ . The tasks to be computed by the 3 workers are $[eqn]$ , $[eqn]$ , and $[eqn]$ , respectively. The workers are denoted as $[eqn]$ , and the input files are denoted as $[eqn]$ . Before starting the task, the input files are assigned as follows: $[eqn]$ to worker 12, $[eqn]$ to worker 13, and $[eqn]$ to worker 23.

3.3.1. Map Phase

In the map phase, each worker maps each assigned file into 3 IVs. Specifically:

• Worker 12 owns $[eqn]$ but lacks $[eqn]$ .
• Worker 13 owns $[eqn]$ but lacks $[eqn]$ .
• Worker 23 owns $[eqn]$ but lacks $[eqn]$ .

3.3.2. Shuffle Phase

After the map phase, the shuffle phase begins, where workers exchange IVs via the APs. Each worker encodes the IVs $[eqn]$ into $[eqn]$ using a randomized Gaussian coding scheme to ensure reliable transmission over the wireless channel.

Time Slot 1: Workers 12 and 13 send $[eqn]$ and $[eqn]$ to AP 1 simultaneously. AP 1 receives the message $[eqn]$ , completing the uplink process. AP 1 then broadcasts $[eqn]$ to workers 12 and 13, who receive $[eqn]$ and $[eqn]$ , respectively.Time Slot 2: Workers 12 and 23 send $[eqn]$ and $[eqn]$ to AP 2 simultaneously. After the uplink and downlink phases, workers 12 and 23 receive $[eqn]$ and $[eqn]$ , respectively.Time Slot 3: Workers 13 and 23 send $[eqn]$ and $[eqn]$ to AP 3 simultaneously. After the uplink and downlink phases, workers 13 and 23 receive $[eqn]$ and $[eqn]$ , respectively.

3.3.3. Reduce Phase

In the reduce phase, each worker decodes the missing IVs:

• Worker 12 solves $[eqn]$ and $[eqn]$ from $[eqn]$ and $[eqn]$ using its own $[eqn]$ and $[eqn]$ .
• Worker 13 solves $[eqn]$ and $[eqn]$ from $[eqn]$ and $[eqn]$ using its own $[eqn]$ and $[eqn]$ .
• Worker 23 solves $[eqn]$ and $[eqn]$ from $[eqn]$ and $[eqn]$ using its own $[eqn]$ and $[eqn]$ .

After obtaining all the required IVs, each worker computes its assigned output function.

In this example, the computation load is $[eqn]$ . Assuming each IV $[eqn]$ has a size of t bits, the uplink and downlink loads are calculated as $[eqn]$ .

4. Conclusions

In this paper, we studied the coded distributed computing for $[eqn]$ -combination networks and provided a comprehensive framework for constructing CDC scheme from the view point of linear algebra. Then, by using CPDAs, we proposed two classes of CDC schemes for the combination network. Finally by comparisons, we showed that the proposed schemes have lower communication loads than the baseline schemes.