Multi-Point Synchronization for Fog-Controlled Internet of Things

TL;DR

This paper introduces a fog-based controller architecture for synchronizing large IoT networks, employing multi-point scheduling algorithms that handle disconnections and failures, with practical implementation and evaluation.

Contribution

It develops novel multi-point synchronous scheduling algorithms for IoT, addressing disconnections and failures, and demonstrates their effectiveness through experiments and implementation.

Findings

Algorithms effectively handle disconnections and failures.

Time-based redundancy improves synchronization reliability.

Implementation in JAMScript shows practical viability.

Abstract

This paper presents a fog-resident controller architecture for synchronizing the operations of large collections of Internet of Things (IoT) such as drones, Internet of Vehicles, etc. Synchronization in IoT is grouped into different classes, use cases identified and multi-point synchronous scheduling algorithms are developed to schedule tasks with varying timing requirements; strict (synchronous) and relaxed (asynchronous and local) onto a bunch of worker nodes that are coordinated by a fog resident controller in the presence of disconnections and worker failures. The algorithms use time-based or component-based redundancy to cope with failures and embed a publish-subscribe message update scheme to reduce the message overhead at the controller as the number of workers increase. The performance of the algorithms are evaluated using trace-driven experiments and practicability is shown by implementing the time-based redundancy synchronous scheduling algorithm in JAMScript -- a polyglot programming platform for Cloud of Things and report initial findings.

Tables

Table 1. TABLE I: Algorithm notations.

Symbol	Description
$T_{curr}$	current task to be scheduled
$T_{rbq}$	ratio-based quorum check task
$T_{cbq}$	cluster-based quorum check task
$T_{update}$	status update task
$t_{avail}$	available time of a worker
$\lambda$	delay before re-attempting quorum
$\delta$	predicted quorum check time

Full text

I Introduction

IoT come in many shapes and sizes. Because each IoT component will be handling a part of a problem domain, cooperative computing among the IoT components is essential for solving many interesting problems [1]. For example, consider a situation where a swarm of drones is required to take a snapshot of a search area. A synchronization mechanism is required to ensure that the autonomous drones capture a specified part of the search area at the same time. If the drones do not follow strict timing alignments, it will lead to a poor reconstruction of the search area. As another example take an autonomous car, the car could form a swarm with other cars and road side units. To drive successfully, the car needs assurance that data is collected within a short time interval by the sensing and processing components of the swarm. The car thus needs support from the swarm to ensure that its tasks are run simultaneously across the different constituents.

Cooperating IoT need to run tasks of the application in a coordinated manner. One such coordination is running the tasks at the same time. For instance drones lifting a workload need to exert the force at the same time so the effort adds up. To run the tasks at the same time, there is a need for synchronization primitives that would launch the tasks with the start times lined up at all IoT devices. Just having synchronized clocks [2, 3] is not sufficient because even with an earlier agreed start time, the IoT devices could fail, disconnect from the swarm [4], or be busy with a prior task. With tighter synchronization schemes, fine-grained tasks can be run across the cooperating IoT.

In this paper, a controller-based scheme for synchronizing the execution of the tasks across a cooperating set of IoT devices is presented. With a controller-based scheme, one of the issues is the location of the controller itself. The recent emergence of fog computing as a major complementary technology to IoT is making it an ideal candidate to host our controller. Fogs are placed closer to the IoT devices they serve, placing the controller on a fog makes it accessible with the lowest possible latency from all IoT devices in the cooperating set [5].

The primary focus of this paper is on developing such synchronization schemes that guarantees the desired quality of synchronization, $QoS_{ync}$ subject to the conditions under which synchronization is deemed successful. The synchronization scheme has to ensure that the constituent tasks of an application (asynchronous, synchronous and local tasks) are scheduled onto a bunch of cooperating devices while ensuring that the $QoS_{ync}$ is met.

The major contributions of this work are as follows:

A hierarchical system with a controller-worker model for running tasks with strict coordination among cooperating IoT components is presented. 2. 2.

Two redundancy-based dynamic synchronous scheduling algorithms with different synchrony requirements to handle synchronous, asynchronous and local tasks are developed. 3. 3.

The practicability of the synchronous scheduling scheme is shown by implementing the time-based redundancy algorithm in JAMScript – “a programming language and framework developed for Cloud of Things [6]”.

Hereafter, IoT devices are referred to as workers and sync is used to refer to “synchronization” or “synchronous”. The rest of this paper is structured as follows. A background and motivation for this paper is given in Section II. The system architecture comprising of the node, application and task models is presented in Section III. The algorithms and experimental results are presented in Sections IV and  V respectively. Section VI provides details on implementation and the observations made. The related work is given in Section VII.

II Background and Motivation

Synchronization is a well studied topic in computer science. In gang scheduling [7] and coscheduling  [8] where tasks of a parallel job are scheduled and executed at the same dedicated node, synchronization is achieved by using busy waiting. In parallel computing models such as the bulk synchronous parallel (BSP) model [9] where the execution model is broken into computation and communication super steps, synchronization is achieved by using barriers. Barrier synchronization involves processes stopping at the barrier until all other processes reach the barrier [10, 11]. Thus, faster processes have to wait for slower processes, with the bottleneck being the slowest process. Time slotting has been adopted as a way of achieving synchronization in wireless sensor networks [12, 13]. Dedicated synchronization time slots are chosen and devices attempt to perform synchronization tasks only in the dedicated time slots. Time slotting suffers from stragglers as with barrier synchronization. Slow workers will miss the dedicated synchronization slot and will result in the synchronization failing.

Synchronization schemes in distributed systems cannot be directly applied to IoT due to the following factors. (i) Node connectivity could be highly unstable in IoT due to mobility and disconnections unlike distributed systems where stable connection exists among nodes. Synchronization schemes in distributed systems are developed based on this assumption and are thus not suitable for highly dynamic systems. (ii) The network topology rapidly changes due to nodes joining and leaving in IoT unlike distributed systems. (iii) There is an interaction with physical things that have real-time window constraints in IoT unlike traditional distributed systems which do not necessarily affect real world systems. (iv) Nodes in IoT could be highly heterogeneous e.g., sensors, mobile phones, cars etc., unlike distributed systems where nodes have similar characteristics.

II-A Definition and Taxonomy of Synchronization in IoT

Synchronization in IoT in the scope of this work is defined as the coordination of a set of IoT nodes to harmonize on the execution of a task at the same point or at multiple points in time.

Classification of synchronization in IoT is done based on unique attributes and requirements of synchronization in IoT. The taxonomy of synchronization in IoT is shown in Fig. 1 and the classification is as follows.

II-A1 Total number of controllers

The number of controllers in an IoT system could be single or many depending on the scope of the system. In the case of a single controller, all workers must be connected to the controller to be part of the system. In a multi-controller system, workers connect to the controller that is closest to them or assigned to their physical vicinity. Workers can change controller membership as they move around.

II-A2 Number of controllers at synchronization point

Synchronization could be at a global scale or local scale. In a global synchronization, all the controllers in the system must ensure that the devices under them synchronize their activities on the given sync task. In a local synchronization, only a subset of controllers that are affected by the synchronization are involved in the synchronization process.

II-A3 Synchronization point time alignment

All the controllers participating in the synchronization can either orchestrate the devices under them to start the execution of the sync task at the same time or at different times. Thus, a single controller system can only permit an aligned sync point time while a multi-controller system can permit both an aligned and a non-aligned sync point time.

II-A4 Node participation

Some synchronization tasks require either all or some specified number of the devices connected to a controller to partake in the synchronization process. The participation of devices is dependent on their availability to run the sync task and the requirements of the sync task itself.

II-A5 Node types

Synchronization in IoT is affected by node types. Nodes could be homogeneous, having similar attributes or heterogeneous, having different attributes.

II-A6 Node mobility

Mobility is an important factor affecting synchronization in IoT. Nodes could be mobile, such as hand-held devices, cars etc, or fixed such as nodes placed on lamp posts.

II-A7 Quorum requirement

The conditions at which synchronization can occur are either having at least the required fraction of nodes available to run the synchronous task (ratio-based quorum) or having the desired representation per group of clustered nodes (cluster-based quorum). The $QoS_{ync}$ is defined by the quorum requirement conditions.

II-A8 Synchronicity

In atomic synchronization, a sync task is executed or not depending on the quorum success or failure. In eventual synchronization, the sync task is run regardless of whether the nodes are synchronized, synchronicity is expected to increase with time.

II-B Synchronization Use Cases

The use cases for synchronization in IoT are identifies as follows:

II-B1 Capacity pool

IoT and other smart devices are usually limited in their computing and sensing capabilities. Thus, there is a need for cooperation among several devices to solve a much bigger problem than can be solved by an individual device. The devices must pool their resources together in a coordinated manner in order to be successful. An example is autonomous drones pooling their lifting capacities to carry a large package from one point to another.

II-B2 Data capture synchrony

Data synchronization [14] is an important problem in IoT data acquisition. Using synchronous tasks for capturing data allows us to precisely control the relative timing relations among the data points. In data capture synchrony, different subset of nodes can be made to synchronize at different points in time thus achieving some level of data ordering.

II-B3 Resource usage synchrony

IoT devices consume resources (e.g. power) for their operation; therefore, there is a need for subset synchronization to sequence the operating order of the IoT devices to minimize the maximum resource usage profile. Take for example a smart lighting system consisting of a large number of bulbs, to light up (cover) a particular area, only a subset of the bulbs need to be turned on at the same time. Synchronization can be used to incrementally change the lighting intensity or maintain a constant lighting intensity.

II-C Example Deployment Scenarios

Three application scenarios where synchronization among devices is of high importance are given.

II-C1 Drone Delivery System

Drone transportation [15] is becoming increasingly popular and lots of research and industrial effort is going into using drones as a means of transporting humans, goods (e.g. Amazon and UPS), vaccines and other products. In goods transportation, rather than use a large drone, smaller drones could be deployed and coordinated to carry heavy goods from one facility to another.

Drone transportation is an example of capacity pooling where the required number of drones is pooled from the available drones with component-based redundancy included to cater for failures. Time, location and redundancy are important factors in drone transportation. A certain number of drones are needed at the start location before a lift could commence. Predictive models can be used to estimate the availabilities of the drones at different locations and times which can be used to improve the overall transportation schedule.

II-C2 Smart Cars and Self Driving

A smart (autonomous) car carries enormous amount of computing, storage and sensing capacities [16]. With fast wireless networks and fog computing, smart cars can share their capacities with other cars or with the road side infrastructure and vice-versa. Such a swarm of cars will have significantly augmented capabilities; that is, a smart car could have more capabilities (i.e., higher level autonomy [17]) on a smart highway (SH) than what is capable of on a normal highway.

Smart cars need up-to-date information about the status of the road (presence of other cars and the road condition) and other driving conditions (such as weather) to drive safely. The SH can be divided into segments with each segment providing virtual resources to the swarm of smart cars within its range. The shared pool of resources including video cameras, pressure sensors, and speed monitors need to operate synchronously to tackle the tasks in real time without creating backlogs (i.e., a capacity pool use case for multi-point synchronization). The challenge here is to deal with coalitions that are short-lived (i.e., coalitions created and destroyed as cars move by) with low synchronization overhead.

II-C3 Bridge Health Monitoring

This is an example of the data capture synchrony use case. Strain measurement at bridge joints and other important points in the structure is required to maintain a close watch on the health of a bridge structure. To make high quality measurements, it is necessary to coordinate the data capture operations such that they are made when the loading is at a particular configuration. The loading configuration would be measured by the position of the vehicles on the bridge at that instant and their weight.

The most accurate way of doing such a measurement is to actuate all involved devices (sensors and vehicles) to run the measurement function at the same time instant. If different devices run the measurement function at different time points, a complex reconstruction procedure need to be executed to determine the concurrent loading. The fine-grained measurements are only of interest while the vehicles are on the bridge. That is, they do not need to continue to take fine grained position measurements and report them when they are not on the bridge.

III System Architecture

In the system, nodes are organized into a multi-level controller-worker tree as shown in Fig. 2 with the cloud at the topmost level of the tree and worker nodes at the leaves of the tree. Workers could be fixed or mobile. Fogs are located between the cloud and workers. The cloud controller has a global view of the entire system while the fog-level sub-controllers have limited view but are located closer to the workers. This architecture is suitable for achieving synchronization in IoT because it permits worker nodes to join, leave or move around, thus, changing membership one sub-controller to the other. The fog-level controllers provide localized and low latency services to the worker which is necessary for getting synchronization in IoT.

Multiple levels of sub-controllers could exist between the cloud and the workers, the sub-controllers could be at the fog and device level. Tight clock synchronization is assumed across all the levels in the tree by leveraging the hierarchical architecture and the recent advancements in clock synchronization schemes [18]. Our controller-worker model is inspired by systems such as multi-robot systems [19], software defined networking (SDN) [20] and fog computing systems.

III-A Node Model

Both the controllers and workers in our system follow a single-threaded execution model (i.e., they can only run a single task at a time). Worker nodes are isolated from each other and thus, no direct link exists between them. Workers only communicate with the sub-controller or controller to which they are connected. Communication between a worker and controller is bi-directional, that is data can flow both ways. The scope of the task being run determines what level of controller the workers communicate with. A global scope will require communication to exist between the main controller (cloud-level controller) and workers either directly or through sub-controllers. A non-global scope involves workers communicating with the fog-level controller while a much more localized scope involves workers communicating and working with only the device-level controller.

A heterogeneous system is assumed where worker nodes have varying processing and computational capabilities, thus, worker nodes are expected to have different execution times for the same task. Due to network disruptions, mobility or node failures, disconnections can occur between worker nodes and controllers. Disconnected workers can rejoin the system by connecting to a controller and new workers can only join the system if they connect to a controller.

III-B Application and Task Model

An application written for this model consists of remote and local function calls. The remote function calls invoked by the controllers on sub-controllers and workers are downcalls while calls by the workers on controllers at any level (i.e., one or more of the cloud, fog, device levels) are upcalls. For instance, the root controller can downcall to the fog-level or device-level.

The upcalls and downcalls can either be synchronous or asynchronous as shown in Fig. 3. A synchronous call blocks the calling node until the remote execution completes while an asynchronous call is non-blocking. Because the upward and downward calls can involve multiple workers, the synchronous execution of the calls is not simple. In particular, the conditions under which the group would provide a valid execution for a synchronous call needs to be defined.

When a controller makes a synchronous downcall, workers individually sign up for executing the function. Once the controller receives commitments from sufficient number of workers to run the function, the controller will proceed with running the function on the workers. The requirement is to start the execution precisely (assuming the underlying clocks are synchronized) at the same time across all the workers. This is a hard problem because even the signed-up workers can become available to execute the function at different times. Therefore, the function execution needs to be scheduled across the workers such that the start time skew and idle times at the workers are simultaneously minimized.

Tasks are classified based on function calls into the following categories.

Controller-to-worker asynchronous call: This is a call from a controller to its worker nodes to run a task without having strict timing conditions, that is, the controller does not wait during this period. This task is denoted as $c2w_{a}$. 2. 2.

Controller-to-worker synchronous call: The controller sends a command to all its worker nodes to start executing a task at the same point in time. The controller waits until all worker nodes have finished executing the task and returned an output. This task is denoted as $c2w_{s}$. 3. 3.

Worker-to-controller asynchronous call: This is a call from a worker node to one of its controllers to run a task. The worker does not wait during this period and continues with its own execution plan. This task is denoted as $w2c_{a}$ 4. 4.

Worker-to-controller synchronous call: The worker node waits for an acknowledgement from the controller that it has finished executing the task associated with the call. This task is denoted as $w2c_{s}$ 5. 5.

Self calls: This is a call from a node to itself. It could be either a worker node calling itself or a controller calling itself. A self triggered worker task is called a local worker task and is denoted as $w_{l}$ while a self triggered controller task is denoted as $c_{l}$.

IV Algorithms

Two dynamic synchronous scheduling algorithms with redundancy built into them for scheduling tasks with varying time requirements onto a bunch of IoT nodes are developed. The unique aspects of the algorithms are an adapted publish-subscribe status update scheme, quorum checking, redundancy and the local scheduler which are explained in the following subsections. The main focus of the algorithms is to ensure that synchronous tasks are scheduled to run at the same time across all workers. Tasks are gotten from the application graph and arrive in order of precedence such that a task depending on output data from a previous task cannot be spawned until the previous task has finished executing.

One input to the synchronous scheduling algorithms is a set of available tasks. The algorithms are expected to output a schedule of tasks that minimizes the overall execution time and ensures that the desired level of synchronization is met with respect to the quorum requirements. In addition to the notations introduced in Section III, the following notations are used in the algorithms. A description of symbols used in this paper is shown in Table I.

IV-A Status Update Scheme

Workers need to update the controller of their availability to partake in synchronization upon reaching a synchronization point. The controller processes this information serially due to its single-threaded nature. Thus, the message complexity of the update sending process scales up linearly with the number of workers; thus, increasing the time required to achieve synchronization significantly.

To mitigate this problem, the popular publish-subscribe scheme is adapted to reduce the number of update messages sent to the controller. Worker nodes are grouped into clusters and each cluster is assigned a local broker, where all the workers publish their availabilities. They also subscribe to the broker to know the peer availabilities and when a worker detects that the required number of peer workers are present, it will publish a group availability message at the broker. The controller subscribes to the group availability message but not the not local availabilities. So, it ends up processing far less with the broker.

IV-B Quorum Types

Quorum checking is done by workers to probe the controller on whether the required conditions for proceeding to run the synchronization task are met, thus providing some $QoS_{ync}$. Two types of quorum conditions are considered. One based on ratio of workers available and the other on cluster representation. A worker is said to be available if it has finished executing its previous task and is physically present to run the next task. The two types of quorum checking are ratio-based (launched on workers by running $T_{rbq}$) and cluster-based (launched on workers by running $T_{cbq}$) quorums, both of which are presented in the taxonomy in Section II-A7.

IV-C Synchronous Scheduling Scheme

As seen in Fig. 4, the controller sends a sync call (syncCall()) to the workers on getting to a synchronization point. The workers update the controller of their status (pushStatusUpdate()) either using a naive scheme where all workers send update messages to the controller or using the adapted publish-subscribe update scheme. The pseudocode for the algorithm is shown in Algorithm 1.

The controller computes the predicted quorum check time $\delta$ based on the status update from workers and sends the time to local schedulers on workers (sendQuorumCheckTime($\delta$)). The local scheduler on a worker tries to minimize wait time by comparing the available time of the worker with $\delta$ and estimates if a local task (provided there is one in the local task queue) can fit within the gap as shown on Line 6 of Algorithm 1. The local scheduler (localSchedule($\delta)$) is an optimization scheme to minimize the idle time on workers that become available earlier than other workers as shown in Algorithm 2. Asynchronous and local worker tasks are scheduled as they become the current task at the earliest available times on workers.

The two types of the dynamic synchronous scheduling algorithm are the time-based redundancy where synchronization is attempted based on waiting and a capped number of retries, and the component-based redundancy where at least a certain number of workers within a group are expected to be present before quorum can be passed.

IV-C1 Synchronous Scheduling Algorithm with Time-Based Redundancy

Here, the synchronization degree represents the ratio of workers that must be available before the synchronous task can be run. Whenever the ratio-based quorum check task $T_{rbq}$ is scheduled (Line 9), the controller computes the ratio of available workers. If the ratio of available workers is greater than or equal to the expected synchronization degree (Line 12), the controller computes the expected synchronous task start time and the synchronous task is scheduled.

If the synchronization degree is not met and there are more retries left (Line 14), execution is delayed for time $\lambda$ and the process is retried. However, if the synchronization degree is not met and there are no more retries left, the synchronous task is failed (Line 17). An example where time-based redundancy is useful is in bridge health monitoring. If while trying to take measurements a failure occurs, the application can proceed with its processing and repeat the bridge strain measurement at a later point in time.

IV-C2 Synchronous Scheduling Algorithm with Component-Based Redundancy

Here, workers are always part of a logical cluster as they move around in the system (e.g., vehicles). At a synchronization point, after the update of the workers’ status at the controller (Line 5) and the triggering of the local schedulers (Line 7), the workers run the quorum check task ($T_{cbq}$). The controller computes and checks whether the required level of redundancy is met by each cluster (Line 19-20). This computation is done by comparing the number of workers available in each cluster with the expected number of devices per cluster. In the event of a successful quorum, the synchronous task is scheduled to run. If the desired level of redundancy is not met, the synchronous task is failed (Line 23).

In this algorithm, a synchronous task is marked as successfully completed if and only if at least the required number of workers per cluster returns a result after executing the synchronous task. Thus, the synchronization result is abstracted at the cluster level. If the desired redundancy in output is not met, the synchronous task is considered to have failed. An example of where component-based redundancy is useful is in the drone transportation scheme. The drones are expected not to perform the transportation task if the desired number of backup drones are not available.

IV-D Local Scheduler

The local scheduling is an optimization scheme developed to minimize the waiting of workers that get to the synchronization point earlier than other workers. Since the workers do not communicate directly with each other but only with the controller, workers on getting to a synchronization point have no information about other workers. So, by running a local scheduler, a worker compares its current time with the predicted finish time $\delta$ across all workers as shown in Algorithm 2.

The local scheduler would run a local task if the gap between the predicted availability of the other workers $\delta$ and the local node’s availability $t_{avail}$ is greater than the local task’s length. Otherwise, the local scheduler would not schedule any local task and let the worker sit idle.

V Experiments and Results

First, experiments are performed to measure the impact of controller location (i.e., fog or cloud) on synchronization. Then, further experiments are conducted to measure specific attributes of the performance of the proposed synchronization algorithms. Finally, the performance of the proposed synchronization algorithms are evaluated by comparing them with barrier synchronization [10, 11] and time slotted synchronization [12].

V-A Impact of Controller Location on Synchronization

In this experiment, traces from the OpenCloud Hadoop cluster from Carnegie Mellon University Parallel Data Lab 111http://ftp.pdl.cmu.edu/pub/datasets/hla/dataset.html are used. The workload is split into short tasks ranging from 0.5s to 4s and long tasks ranging from 5s to 12s. The controller-worker delay is varied from 5ms to 500ms. Performance is measured using synchronization rate (SR) which is the number of synchronizations per unit time.

From Fig. 5, it can be observed that a task graph consisting of short jobs has much higher SR compared to a task graph consisting of long tasks. Also, increasing the controller-worker delay from 5ms to 500ms has very little on impact on the task graph with long tasks compared to short tasks because the long tasks take a significant portion of the overall runtime thus minimizing the impact of the controller-worker delay. A task graph with short tasks consisting of synchronous, asynchronous and local tasks has a higher SR compared to one consisting of only synchronous tasks. This is because with only synchronization tasks, status updates and quorum checking needs to be performed at each synchronization point therefore adding more overhead to the overall runtime. Having controllers closer to the workers increases the SR for short running tasks, thus, making the case for fog-resident controllers as opposed to cloud-resident controllers.

V-B Configuration of Synchronization Experiments

The following configurations are used to define the system, applications and environmental parameters. The wide range of parameter variation allows for a higher degree of exploration into many aspects of the system. Measurement traces from experiments using Dropbox between the period of 28th June to 3rd July 2012 [21] are used as the task dataset. The execution times vary with a minimum time of 23$s$ and a maximum time of 269$s$.

To model the mobility of worker nodes, the Shanghai (China) taxi GPS report for Feb 20, 2007 [22] is used. The report consists of 4316 taxis reporting their location, speed, angle of movement and occupancy at given intervals over a period of 24 hours. Each taxi has a unique identification number. The taxi traces are pre-processed and only the location and timestamp details are extracted. The taxis upload their details at irregular intervals in trace, varying from 15s to 63s. Re-sampling is done at 30s and the position of the taxis are gotten at regular intervals, thus having a total of 2880 time points.

The parameters used in the simulations are as follows. (i) Synchronization degree: ratio of the total machines required to pass quorum. (ii) Minimum cluster size: the minimum number of workers that must be present in a cluster before it can be formed. (iii) Wait time: The amount of time that should elapse before attempting quorum checking again. (iv) Quorum retries: the maximum number of times quorum checking is allowed. (v) Worker size: The maximum number of workers that can be present in the system at any point in time. (vi) Number of clusters: The maximum number of clusters that can be formed at any point in time. (vii) Prediction accuracy: A measure of how accurately the predictor predicts the finish time of tasks across all workers prior to the synchronization point.

V-C Default Parameter Values and Measurements

The following parameters are fixed in the simulations unless otherwise stated. The number of independent runs of each simulation is $100$ while each task graph is continuously run in each simulation for $200$ times, the communication cost between machines is set to $200ms$, status update cost is set to $1s$, synchronization degree is set to $0.7$ and $\lambda$ set to $20s$. Workers randomly fail with a probability of $0.1$ after each task. The probability of a new machine joining is set at $0.1$, but machines can only join at the start of the execution of a new run of the task graph. This is done to ensure that joining machines will have all the necessary data required to run all tasks down the task graph.

Task graphs consist of $30$ tasks in total with varying number of synchronous tasks. $10$ task graphs are used to represent different applications in our simulation runs. Heterogeneity among the worker nodes is introduced by making the execution time of a task on multiple workers follow a Gaussian distribution.

The following parameters are measured in the simulations. (i) Runtime: the time taken for a single run of a task graph. It is gotten by dividing the total execution time of a simulation by the number of runs. In our results, the execution time is normalized by the number of sync points. (ii) Extra quorum attempts: the total number of times the quorum check process was attempted after the initial attempt at all synchronization points. (iii) Failed sync tasks: The total number of sync tasks that failed after exceeding the total number of quorum retries or due to incomplete results from clusters.

V-D Experiments, Results and Discussions

V-D1 Scalability of the adapted publish-subscribe update scheme

On getting to a synchronization point, workers update their associated controller of their availability for running synchronous tasks. The updates are serially processed by the controllers due to their single-threaded nature. This can pose a serious bottleneck problem as the number of workers increase. To alleviate this, a publish-subscribe mechanism is adapted. The benefits of the mechanism for are shown in Fig. 6 while varying the number of workers from $10$ to $4000$.

Fig. 6 shows the runtime per synchronization point for the publish-subscribe and all-worker update methods for the ratio-based quorum checking. From the graph, it can be observed that as the number of workers increase the runtime per synchronization point increases at a similar rate with respect to the number of workers for the all-worker update while for the publish-subscribe update there is no significant increase in the runtime per synchronization point as the number of workers increase. This is because regardless of the number of workers in the system, only a constant number of update messages is being sent to the controller in the publish-subscribe update method.

V-D2 Component redundancy

In component-based redundancy, the workers are grouped into clusters. To reach a quorum to execute a synchronous task, at least a given number of workers must be available in each cluster. The synchronization task is considered successful if at least one worker from the number of workers in the cluster completes the execution of the task and returns the expected output to the controller, otherwise, the synchronous task is failed.

The minimum required number of worker(s) per cluster is varied from $1$ to $4$, the synchronization task failures are measured and shown in Figs. 9 - 9. Figs. 9 and 9 show the synchronization tasks failure percentage due to failed quorum and incomplete results from clusters, respectively. The percentage of synchronization task failure due to failed quorum and incomplete results from clusters both decrease as the minimum cluster size increases. This is due to the fact that the probability of having workers show up during quorum increases as the minimum cluster size increases.

V-D3 Impact of finish time prediction accuracy

The local scheduler uses the predicted availability of the workers to determine whether it can schedule a local task before starting the synchronous task. In Fig. 9, the impact of finish time prediction accuracy on the runtime per synchronization point and the synchronization task failure rate with component-based redundancy and publish-subscribe update schemes are measured.

When the prediction accuracy is $100$%, it was observed that the smallest runtime per synchronization is $4940s$ and the smallest synchronous task failure rate of $8$%. While at a finish time prediction accuracy of $80$%, an average of $5730s$ was observed for the runtime per synchronization point and a $30$% synchronization task failure. The runtime per synchronization point increases by $16$% while the percentage synchronization task failure increases by $263.3$% as the finish time prediction accuracy reduces from $100$% to $80$%. This shows the high impact that the finish time prediction accuracy has on the success of the synchronization task.

V-D4 Performance Evaluation

The performance of the proposed synchronization schemes (time-redundant and component-redundant synchronization) is evaluated by comparing them with barrier synchronization and time slotted synchronization. In barrier synchronization, on getting to a sync point, workers send update message to the controller and wait until a signal is received from the controller saying that they can proceed to run the sync task. The condition for proceeding with the barrier execution is that all the workers must reach the sync point.

In time slotted synchronization, the workers’ executions are split into time slots. Dedicated synchronization time slots are chosen with the hope that workers will be available to run the sync task at the specified time slot. The dedicated synchronization time slots are chosen by fixing the slots at $\mu+1.5\sigma$ (accounts for a 86.6% accuracy), where $\mu$ is the average execution time and $\sigma$ is the standard deviation.

Figs. 10 - 13 show the runtime per synchronization point and percentage synchronization task failure for the synchronous scheduling algorithm with the proposed time-redundant and component-redundant synchronization algorithms, barrier synchronization and time slotted synchronization, respectively. The minimum cluster size for component redundancy is fixed as 3 while comparing the synchronization schemes. It can be observed from Figs. 10 and 13 that barrier synchronization has the highest runtime, followed by the time-redundant synchronization algorithm and then time slotted synchronization. Barrier synchronization takes longer because faster workers need to wait for stragglers at the barrier and cannot proceed until the slowest worker reaches the barrier. This is unlike the time-redundant synchronization algorithm that was proposed here, where the synchronization point is moved dynamically depending on the availabilities of workers and also, the local scheduling mechanism is used to minimize wasted work cycles due to waiting. Time slotted synchronization is faster because there are dedicated synchronization slots that are not moved regardless of workers availabilities.

Figs. 13 and 13 show the percentage synchronization task failure due to failed quorum for the proposed time-redundant and component-redundant algorithms, and the time slotted synchronization scheme. The percentage of sync task failures for the time slotted synchronization scheme is higher than that for the proposed time-redundant and component-redundant synchronization algorithms as shown in Figs. 13 and 13 respectively. In Fig. 13, it can be observed that the percentage of sync task failures reduces as the number of workers increase. This is because there are more devices per cluster and thus, there are more redundant devices which increases the chances of reaching quorum. Time slotted synchronization have higher sync task failure rates because the synchronization slots are fixed, and when there are straggling workers synchronization cannot proceed.

VI Implementation and Observations

VI-A Implementation Details

The time-based redundancy algorithm (shown in Section IV-C1) is used to implement multi-point synchronization in JAMScript. JAMScript is a polyglot language; it brings together C and JavaScript with some additional constructs needed to implement the programming model. The C side implements the worker side of the programming model and JavaScript side implements the controller side. This means the C side runs at the lowest level of the system. For instance, embedded devices (e.g., sensors and actuators) will be running the C component. This allows us to target memory constrained devices (planned for the future). The JavaScript running in a NodeJS runtime is responsible for implementing the controllers. The NodeJS based controller can run from low power devices to all the way to cloud servers.

This implementation is achieved in two stages. In the first stage, a single-point synchronization scheme is realized, where only one controller orchestrates the operation of all workers. At this time, the single-point synchronization is basic and lacks the local scheduler optimization to minimize the idle time in the workers. Therefore, the local, asynchronous, and synchronous task order is kept the same as that specified in the program. In the second stage, the single-point synchronizer is used to create a multi-point synchronizer by using a hierarchy of controllers. For instance, by placing a controller at C1 the operations are synchronized across all the workers in Fig. 2. Alternatively, if the controllers are placed at F1 and F2, which themselves are coordinated by a super controller at C1, then the workers can be synchronized in two sub groups: G2 and G3.

VI-B Example JAMScript Program Snippets

Listing 1 shows the controller side of the example JAMScript program. The trytellid() function is responsible for calling the synchronous function tellid() on the worker nodes. The trytellid function is a conditional function that only runs at the fogs.

That is, when the given JAMScript program is run in a resource configuration shown in Fig. 2, the trytellid runs only in Fogs F1 and F2. F1 would synchronize G2 and F2 would synchronize G3. However, workers in G2 and G3 would not be synchronized with each other; thus, presenting multi-point synchronization.

VI-C Preliminary Experiments and Discussions

Fig. 14 shows the performance of single-point synchronization of a JAMScript program when it is deployed in Docker containers.

A task graph with synchronous and asynchronous tasks whose execution times on nodes follow a normal distribution with a mean of $1s$ and a variance of $0.25s$ is used. A normal distribution with a mean of $25ms$ and a variance of $10ms$ is used to represent the network delay. The number of nodes is varied from $2$ to $200$.

As part of the future work, an optimized implementation that should reduce the overhead using publish-subscribe and local scheduling will be implemented. It is worthy to note that while JAMScript is used to show the feasibility in implementing the synchronous scheduling algorithms, other controller-centric programming frameworks can be used to implement the algorithms.

VII Related work

Biologically inspired synchronization schemes have been well studied such as pulse coupled oscillator (PCO) synchronization [23, 24]. Synchronization in PCO is achieved through a consensus among participating nodes. Assumptions made in PCO are periodic pulse, a mesh connectivity and a zero-communication time. A synchronization scheme combining heartbeat synchronization and a firefly inspired model or overlay networks was proposed in [25]. The proposed protocol has two main parts, nodes selecting their peer list and processing a flash message received in order to achieve synchronization.

An emergent broadcast slot synchronization scheme inspired by firefly algorithms was proposed in [24] for Internet of Things (IoT). Each node maintains a time window that it can be awake during a steady state. Nodes in the network go through three states in order to achieve synchronization. The first state is the initialization state where nodes start their random timers and identify their neighbours. The nodes then transition to the synchronization state, here, the nodes coordinate their synchronization error tolerance window with their neighbours using a Pulse Coupled Oscillator (PCO) model with a phase advancement function. A node becomes synchronized if its synchronicity is greater than the synchronization threshold. Thereafter, the node moves into the steady duty cycle state. In this phase, nodes only wake up during their synchronization error tolerance window to exchange messages.

In real time computing systems, synchronization is handled using time-triggered controls where all synchronous activities are executed at some predefined points in time [26]. Synchronized clocks are used to achieve synchronization in the systems by making each node have a common notion of time. Time synchronization schemes [3, 27] have been developed for IoT to allow devices have a common notion of time. In [3], a visible light produced by light emitting diodes (LEDs) are used by devices within that vicinity to synchronize. Synchronization is achieved by allowing a number of LEDs to send out binary signals at the same time and at predefined intervals, and devices synchronize when there is a phase transition. In [27], a time synchronization protocol was proposed to mitigate the effect of temperature change on hardware clocks in IoT networks using time-slotted channel hopping.

Preemptive synchronized scheduling policies [28, 29] have been proposed for synchronization tasks that share common attributes. In [28], a preemptive synchronized scheduling policy was proposed for synchronization tasks that share same mutual exclusion resources under homogeneous processors. Tasks are allowed to share resources based on a mutual exclusion policy. Tasks that share mutually exclusive logical resources are grouped into the same component and have a local scheduler. Whenever a global scheduler decides on a component to be executed, the local scheduler decides which task gets the resource access. In [29], tasks are partitioned based on communication, synchronization and mutex. Synchronization tasks having the same priority are expected to arrive at the synchronization point (SP) at the same time and cannot be preempted. The synchronization task with the maximum execution time before the SP is scheduled to run first, while that with the maximum execution time after the SP is scheduled to run immediately after the SP.

The synchronization-aware algorithm proposed in [30] bundles tasks by size or mutex-sharing and attempts to schedule them onto a processor. Bundles that do not directly fit into a processor are put in a separate queue and sorted based on their cost (the penalty of transforming a local mutex into a global mutex) in increasing order. The bundle with the smallest cost is broken down into pieces with the largest piece determined by the size of the largest possible space in the processors. If the process is not successful, a new processor is added, and the partitioning process is repeated.

In coscheduling [8] and gang scheduling (a stricter form of coscheduling) [7, 31], tasks of a parallel job (or gang) are scheduled and executed at the same pre-defined time slot. Time slotting is a fundamental notion in gang scheduling where all or none of the members of the gang are run at any time slot. The Ousterhout matrix (OM) is a model for dividing the resources in the space and time dimensions and mapping jobs to them [31]. The OM for gang scheduling represents processing nodes as columns and time slot as rows. Synchronization is achieved in coscheduling and gang scheduling using busy waiting [7].

Comparison of this Paper and Related Work

This work focuses on task and controller driven synchronization. There is a combination of local, asynchronous and synchronous tasks that need to be scheduled unlike the firefly-based (PCO) synchronization schemes  [23, 24, 25] where synchronization is on a single flash message. The proposed algorithms guarantee atomic $QoS_{ync}$ through quorum checking. That is, the execution of a synchronous task cannot proceed unless the synchronicity conditions are met. Whereas in firefly-based (PCO) synchronization, eventual synchronization is sought for. Preemption is not allowed in our scheme unlike in co- and gang scheduling  [7, 8] where tasks that have already started execution can be called back. The proposed synchronization schemes are different from barrier synchronization in that there is no need to wait indefinitely for stragglers as synchronization can proceed once the conditions for quorum are met or the necessary precautions taken if quorum fails. The proposed schemes have dynamic synchronization points that can be adjusted based on the availabilities of workers unlike time slotted synchronization where the synchronization points are predefined.

This work extends beyond time synchronization  [3, 27] in that nodes need not only have the same notion of time, but there is a need for the right availability of nodes before executing a synchronous task. Tasks are not bundled to be scheduled together on the same processor in our scheme as in  [30], rather, the same set of tasks are run across all participating nodes. Node disconnections, leaving and joining are catered for by introducing quorum checking to verify the availability and participation of nodes before executing a synchronous task. Fault tolerance is achieved by introducing time and component redundancy.

VIII Conclusions and Future Work

The need for synchronization in edge-enabled IoT is motivated with application scenarios and use cases. A system model is developed for mapping applications with and without synchronization requirements onto an edge-enabled IoT system. Two dynamic scheduling algorithms are proposed for mapping applications with different quorum requirements. The applications can have synchronous, asynchronous and local tasks in them. The tasks are mapped onto a group of single-threaded heterogeneous nodes that can possibly disconnect from each other. The challenge for the synchronization algorithms is to keep synchrony between the task executions despite disconnections.

The synchronization scheduling algorithms use two ideas: time-based and component-based redundancies. It was observed that time-based redundancy is suitable for applications where repeating the task executions is acceptable. Whereas the component-based redundancy is needed for applications that cannot wait for task re-executions. Experiments are conducted to evaluate the performance of the proposed algorithms and to explore the different trade-offs between the two approaches.

Updating the controller with the progress in task executions is important for synchronization scheduling. It was found that using a publish-subscribe update scheme reduces the communication load on controllers; thus, effectively reducing the overall execution times of the synchronous tasks. It was observed that increasing the level of redundancy for component-based redundancy decreases the runtime and reduces the percentage of sync task failures. Likewise, the prediction accuracy of the finish time of tasks on the workers has a significant impact on the runtime and synchronization task failure. The proposed algorithms have shorter runtimes than barrier synchronization and have less synchronization task failures when compared to time slotted synchronization.

One area of future work is applying machine learning for completion time prediction and incorporating that into synchronization scheduling. Another is to handle device mobility across the fogs. In particular, with vehicular clouds, it is possible to have vehicles joining and leaving different fog zones as they travel. The synchronization scheduler needs to control the vehicle to fog associations to minimize synchronization task failures due to mobility.