Fully Distributed Packet Scheduling Framework for Handling Disturbances in Lossy Real-Time Wireless Networks

TL;DR

This paper introduces FD-PaS, a fully distributed packet scheduling framework for real-time wireless networks that ensures quick disturbance response and maintains QoS without centralized control, suitable for industrial IoT applications.

Contribution

The paper proposes a novel distributed scheduling framework, FD-PaS, that enables local decision-making to handle disturbances efficiently in lossy, real-time wireless networks.

Findings

Validated through extensive simulations and testbed experiments.

Demonstrated fast disturbance response with minimal QoS degradation.

Showed scalability and robustness of the distributed approach.

Abstract

Along with the rapid growth of Industrial Internet-of-Things (IIoT) applications and their penetration into many industry sectors, real-time wireless networks (RTWNs) have been playing a more critical role in providing real-time, reliable and secure communication services for such applications. A key challenge in RTWN management is how to ensure real-time Quality of Services (QoS) especially in the presence of unexpected disturbances and lossy wireless links. Most prior work takes centralized approaches for handling disturbances, which are slow and subject to single-point failure, and do not scale. To overcome these drawbacks, this paper presents a fully distributed packet scheduling framework called FD-PaS. FD-PaS aims to provide guaranteed fast response to unexpected disturbances while achieving minimum performance degradation for meeting the timing and reliability requirements of all critical tasks. To combat the scalability challenge, FD-PaS incorporates several key advances in both algorithm design and data link layer protocol design to enable individual nodes to make on-line decisions locally without any centralized control. Our extensive simulation and testbed results have validated the correctness of the FD-PaS design and demonstrated its effectiveness in providing fast response for handling disturbances while ensuring the designated QoS requirements.

Tables

Table 1. TABLE I: Task parameters for the example RTWN.

Task	Routing Path	$P_i$ ($=D_i$)
${\tau }_0$	$V_2\to V_c\to V_5$	9
${\tau }_1$	$V_0\to V_1\to V_c\to V_5$	9
${\tau }_2$	$V_2\to V_c\to V_3\to V_4$	10

Table 2. TABLE II: Summary of important notations and definitions

Notation	Definition	Notation	Definition
$V_j$ ($j=0,1,\dots$), $V_c$	Device nodes and controller node	$t_{n\to r}$,	Slot when ${\tau }_0$ leaves its nominal state
${\tau }_i\left(0\le i\le n\right)$	Unicast tasks	$t_{r\to n}$	and its rhythmic state, respectively
$H_i$	Number of hops of ${\tau }_i$	$t_{sp}$, $t_{ep}$,	Start point, end point, end point candidate
$P_i$ ($D_i$)	Nominal period (deadline) of ${\tau }_i$	$t_{ep}^c$, $t_{ep}^u$	and end point upper bound
$\overrightarrow{P_i}$ ($\overrightarrow{D_i}$)	Rhythmic period (deadline) vector of ${\tau }_i$	$S$, $\tilde{S}$	Static schedule and dynamic schedule
${\chi }_{i,k}$	The $k$-th released packet of task ${\tau }_i$	${𝐕}_{rhy}$	Set of nodes receiving the disturbance information
${\chi }_{i,k}\left(h\right)$	The h-th transmission of packet ${\chi }_{i,k}$	$\Psi \left(t\right)$, $\Phi \left(t\right)$	Set of active rhythmic and periodic packets
${\lambda }^R$	Required e2e packet delivery ratio (for all tasks)	$\rho [t_{sp},t)$	Set of dropped periodic packets
${\lambda }_{i,j}$, ${\overrightarrow{R}}_{i,j}$	E2e PDR value and retry vector of ${\chi }_{i,j}$	${\rho }^{\ast }[t_{sp},t)$	and transmissions within $[t_{sp},t)$
$R_{i,j}\left[h\right]$	Number of trials for $h$-th hop assigned by ${\overrightarrow{R}}_{i,j}$	${\delta }_{i,j}$	PDR degradation of ${\chi }_{i,j}$

Table 3. TABLE III: Slot Timing Information of MP-MAC

Parameters	Value ($\mu$s)	Parameters	Value ($\mu$s)
SlotDuration	10,000	LongGT	2,200
TxOffset	2,120	ShortGT	1,000
TxAckDelay	1,000	PriorityTick	30 to 400
Ext. SlotDuration	10,800	Extended LongGT	3,000

Equations

$$\{t_{ep}^c\}=\{r_{0,k},\forall r_{0,k}\in [f_{0,m+R},t_{ep}^u]\}$$

$$\forall {\chi }_{i,k}\in \Phi (t_{ep}^c),\min \sum {\delta }_{i,k}.$$

$$\min \sum _{{\chi }_j\in \Phi }{l}_j$$

$$\forall {\chi }_{0,i}\in \Psi ,\sum _{{\chi }_j\in \Phi }{ϵ}_j^i\cdot l_j\ge H_{0,i}-a_i$$

$${\lambda }_{i,k}=\prod _{h=0}^{H_i-1}1-(1-{\lambda }_{L_i[h]}^L{)}^{R_{i,k}[h]}.$$

$$\text{emAdjusted TxOffset}=\text{emTxOffset}+\text{(Priority Level)}\times \text{emPriorityTick};$$

Full text

1 Introduction

Real-time wireless networks (RTWNs) are fundamental to many Industrial Internet-of-Things (IIoT) applications in a broad range of fields such as military, civil infrastructure and industrial automation [1, 2, 3]. These applications have stringent timing and reliability requirements to ensure timely collection of environmental data and reliable delivery of control decisions. The Quality of Service (QoS) offered by a RTWN is thus often measured by how well it satisfies the end-to-end (from sensors via controllers to actuators) deadlines of the real-time tasks executed in the RTWN. Packet scheduling in RTWNs plays a critical role in achieving the desired QoS. Though packet scheduling in RTWNs has been studied for a long time, the explosive growth of IIoT applications especially in terms of their scale and complexity has dramatically increased the level of difficulty in tackling this inherently challenging undertaking. The fact that most RTWNs must deal with unexpected disturbances and the lossy nature of wireless links in industrial environments further aggravates the problem.

Unexpected disturbances in RTWNs in general can be classified into internal disturbances within the network infrastructure (e.g., link failure due to multi-user interference or weather related changes in channel signal to noise ratio (SNR)) and external disturbances from the environment being monitored and controlled (e.g., detection of an emergency, sudden pressure or temperature changes). When an external disturbance is detected by a certain sensor node, the workload associated to this sensor node needs to be changed for a certain time duration to more frequently monitor the environment. Many centralized dynamic scheduling approaches have been proposed in the literature, but most of them are designed for handling changes in network resource supply (e.g., [4, 5, 6]). Studies on addressing external disturbances in RTWNs, the focus of this paper, are relatively few. Most of those work rely on centralized decision making and assume reliable network environments. This motivates us to explore a fully distributed framework for handling external disturbances in lossy RTWNs. In the rest of the paper, we simply refer to external disturbance as disturbance.

The challenge of handling disturbances in RTWNs comes from the unpredictability of disturbance occurrence at run time. Specifically, it is generally unknown when/which disturbance will occur and what is the network status at that point (e.g., how many packets have been delivered to their destinations). Since it is computationally infeasible to enumerate all possibilities before the network starts, on-line dynamic scheduling approaches is required to react fast to unexpected workload changes incurred by disturbances.

The existence of lossy wireless links in the industrial environments raises another challenge in handling disturbances in RTWNs. Specifically, the uncertainty of lossy links in the network introduces packet losses with a certain non-zero possibility. Packet loss in a sensing process can significantly degrade the data freshness, and packet loss in a feedback control may lead to system instability and cause safety concerns. Further, if a packet that delivers disturbance-related information is lost, it may cause catastrophe to the system. Thus, most industrial RTWNs require a desired end-to-end Packet Delivery Ratio (PDR), e.g. $99\%$, for all packets running in the system.

In this work, we introduce a fully distributed packet scheduling framework, referred to as FD-PaS, to handle disturbances in lossy RTWNs.111An earlier version of the paper appeared in [7]. FD-PaS makes on-line decisions locally without any centralized control point when disturbances occur. This is achieved by sending the disturbance information only to a subset of all nodes via the routing paths of the tasks running in the network. In such a manner, a broadcast task is no longer needed in FD-PaS for notifying all nodes about the disturbance information, which significantly reduces the response time to handle the disturbance. To ensure this partial disturbance propagation scheme works properly, we need to overcome several challenges. For example, to avoid transmission collision among different nodes with inconsistent schedules, we propose a multi-priority wireless packet preemption mechanism called MP-MAC in the data link layer to ensure that high-priority packets can always be delivered by preempting the transmissions of low-priority packets. Further, to minimize the timing and reliability degradation, we formulate a transmission dropping problem to determine a temporary dynamic schedule for individual nodes to handle the disturbance. We prove that the transmission dropping problem is NP-hard, and introduce an efficient heuristic to be executed by individual nodes locally. Both the MP-MAC design and the dynamic schedule construction method (they jointly comprise the FD-PaS framework) are implemented on our RTWN testbed. Our extensive performance evaluation validates the correctness of the FD-PaS design and demonstrates its effectiveness in providing fast response for handling disturbances.

The remainder of this paper is organized as follows. The related work is dicussed in Section 2 and Section 3 describes the system model. Section 4 gives an overview of the FD-PaS framework. We discuss how to propagate disturbances and avoid transmission collisions in Section 5 and 6, respectively. Section 7 formulates the dynamic transmission dropping problem and presents the method to determine the time duration for handling disturbance. Section 8 discusses the dynamic schedule generation in both reliable and lossy RTWNs. Performance evaluation are summarized in Section 9. We conclude the paper and discuss future work in Section 10.

2 Related Work

Network resource management in RTWNs in the presence of unexpected disturbances has drawn a lot of attention in recent years. Traditional static packet scheduling approaches (e.g., [8, 9, 10]), where decisions are made offline or only get updated infrequently can support deterministic real-time communication, but either cannot properly handle unexpected disturbances or must make rather pessimistic assumptions. Many centralized dynamic scheduling approaches for handling internal disturbances have been proposed (e.g., [4, 5, 6]). Studies on addressing external disturbances are relatively few and mostly rely on centralized decision making. The approach in [11] stores a predetermined number of link layer schedules in the system and chooses the appropriate one when disturbances are detected. However, this approach is either incapable of handling arbitrary disturbances or needs to make some approximation. Both [12] and [13] support admission control in response to adding/removing tasks for handling disturbances in the network. They however do not consider scenarios when not all tasks can meet their deadlines. The protocol in [14] proposes to allocate reserved slots for occasionally occurring emergencies (i.e., disturbances), and allow regular tasks to steal slots from the emergency schedule when no emergency exists. However, how to satisfy the deadlines of regular tasks in the presence of emergencies is not considered.

In recent years, a number of algorithms have been designed for packet scheduling in Time Slotted Channel Hopping (TSCH) networks, in both centralized (e.g. [15, 16, 17]) and distributed manner (e.g. [18, 19, 20]). Most of those approaches, however, assume static network topologies and fixed network traffic which limit their applications in dynamic networks. To overcome this drawback, [21] proposes Orchestra, a distributed scheduling solution that schedules packet transmissions in TSCH networks to support real-time applications. However, Orchestra does not consider real-time constraint, i.e., ignores the hard deadlines associated with tasks running in the network. It only provides best effort but no guarantee on the end-to-end latency of each task.

In [22], a centralized dynamic approach, named OLS, to handle disturbances in RTWNs is proposed. OLS is built on a dynamic programming based approach which can be rather time consuming even for relatively small RTWNs. Moreover, OLS may drop more periodic packets than necessary due to the limited payload size of the packet in RTWNs and thus further degrade the system performance. To overcome the drawbacks of OLS, D2-PaS in [23, 24] proposes to offload the computation of the dynamic schedules to individual nodes locally by leveraging their local computing capabilities, that is, letting each node construct its own schedule so as to achieve better performance than OLS in terms of fewer dropped packets and lower time overhead. However, as observed from the motivating example presented in [7], centralized approaches, including D2-PaS, suffer from long disturbance response time especially in large RTWNs.

Most MAC layer designs for supporting packet prioritization are based on star topology. For example, the wireless arbitration (WirArb) method [25] is designed to use different frequencies to indicate different priorities. It only supports star topology where the gateway keeps sensing the arbitration signals and determines which user has a higher priority to access the channel. [26] studies a similar problem in the context of vehicular Ad Hoc networks. The proposed multi-priority MAC protocol has seven channels, among which one is the public control channel (CCH) for safety action messages and the others are service channels for non-safety applications. The protocol transmits packets of different priorities with optimal transmission probabilities in a dynamic manner. The PriorityMAC [27] proposes to add two very short sub-slots before each time slot to indicate the priority. Four priority levels are defined but only three levels of over-the-air preemption can be achieved. The last priority level is only used for buffer reordering. In PriorityMAC, a higher priority packet indicates the priority in the sub-slots to deter the transmissions of lower priority packets. PriorityMAC is also based on star topology so each device must be directly connected to the coordinator.

A rich set of methods have been designed for RTWNs to improve the reliability of wireless packet transmission over lossy links in most RTWN solutions (e.g., WirelessHART [28], ISA 100.11a [29], and 6TiSCH [30]). [8] proposed a set of reliable graph routing algorithms in WirelessHART networks to explore path diversity to improve reliability. [31, 32] proposed algorithms to allocate a necessary number of retransmision time slots to guarantee a desired success ratio of packet delivery. However, all aforementioned studies focus on packet scheduling in static RTWN settings over lossy links, and cannot be easily extended to handle abruptly increased network traffic caused by unexpected disturbances.

3 System Model

We adopt the system architecture of a typical RTWN, in which multiple sensors and actuators are wirelessly connected to a controller node directly or through relay nodes. (Note that the controller node is for initial network setup and performing control computations. FD-PaS does not need it for making any on-line decision and updating schedules.) We refer to non-controller nodes as device nodes. We assume that all device nodes have routing capability and are equipped with a single omni-directional antenna to operate on a single channel in half-duplex mode. The network is modeled as a directed graph $G=(V,E)$, where the node set $V=\{\{V_{0},V_{1},\dots\},V_{c}\}$ and $V_{c}$ represents the controller node. A direct link $e=(V_{i},V_{j})\in E$ represents a wireless link from node $V_{i}$ to $V_{j}$ with a Packet Delivery Ratio (PDR) $\lambda^{L}_{e}$, which represents the probabilistic transmission success rate on link $e$222Link PDR $\lambda^{L}_{e}$ is usually measured during the site survey and is stable during normal network operations. In case the value of $\lambda^{L}_{e}$ changes significantly, the new value is assumed to be broadcast to all the nodes in the network.. $V_{c}$ connects to all the nodes via some routes and is responsible for executing relevant control algorithms. $V_{c}$ also contains a network manager which conducts network configuration and resource allocation.

We use the concept of task to describe packet transmission from sensor nodes to actuator nodes. Specifically, the system runs a fixed set of unicast tasks \mbox{\mathcal{T}}=\{\tau_{0},\tau_{1},\dots,\tau_{n}\}. Each task $\tau_{i}\;(0\leq i\leq n)$ follows a designated single routing path with $H_{i}$ hops and we use $\overrightarrow{L}_{i}=[L_{i}[0],L_{i}[1],\dots,L_{i}[H_{i}-1]]$ to represent the routing path of $\tau_{i}$. It periodically generates a packet which originates at a sensor node, passes through the controller node (not necessary for FD-PaS but to carry out control computations) and delivers a control message to an actuator. Fig. 1 depicts an example RTWN with three tasks running on 7 nodes and task parameters are given in Table I.

When external disturbances (e.g., sudden change in temperature or pressure) occur, many IIoT applications would require more frequent sampling and control actions, which in turn increase network resource demands. To capture such abrupt increase in network resource demands, we adopt the rhythmic task model [33] which has been shown to be effective for handling disturbances in event-triggered control systems [22]. (Note that our FD-PaS framework is not limited to the rhythmic task model and is applicable to any task model that provides workload changing patterns for handling disturbances.) In the rhythmic task model, each unicast task $\tau_{i}$ has two states: nominal state and rhythmic state. In the nominal state, $\tau_{i}$ follows nominal period $P_{i}$ and nominal relative deadline $D_{i}(\leq P_{i}$), which are all constants. When a disturbance occurs, $\tau_{i}$ enters the rhythmic state in which its period and relative deadline are first reduced in order to respond to the disturbance, and then gradually return to their nominal values by following some monotonically non-decreasing pattern. We use vectors $\overrightarrow{P_{i}}=[P_{i,x},x=1,\dots,R]^{T}$ and $\overrightarrow{D_{i}}=[D_{i,x},x=1,\dots,R]^{T}$ to represent the periods and relative deadlines of $\tau_{i}$ when it is in the rhythmic state. As soon as $\tau_{i}$ enters the rhythmic state, its period and relative deadline adopt sequentially the values specified by $\overrightarrow{P_{i}}$ and $\overrightarrow{D_{i}}$, respectively. $\tau_{i}$ returns to the nominal state when it starts using $P_{i}$ and $D_{i}$ again.

Here we assume that at most one task can be in the rhythmic state at any time during the network operation. To simplify the notation, we refer to any task currently in the rhythmic state as rhythmic task and denote it as $\tau_{0}$ while task $\tau_{i}\;(1\leq i\leq n)$ is a periodic task which is currently not in the rhythmic state. As shown in Fig. 2, when $\tau_{0}$ enters the rhythmic state, we also say that the system switches to the rhythmic mode. The system returns to the nominal mode when the disturbance has been completely handled, typically some time after $\tau_{0}$ returns to the nominal state. Since disturbances may cause catastrophe to the system, the rhythmic task has a hard deadline when the system is in the rhythmic mode while periodic tasks can tolerate occasional deadline misses.

Each task $\tau_{i}$ consists of an infinite sequence of instances. The $k$-th instance of $\tau_{i}$, referred to as packet $\chi_{i,k}$, is associated with release time $r_{i,k}$, deadline $d_{i,k}$ and finish time $f_{i,k}$. Without loss of generality, we assume that $\tau_{0}$ enters the rhythmic state at $r_{0,m+1}$ (denoted as $t_{n\rightarrow r}$) and returns to the nominal state at $r_{0,m+R+1}$ (denoted as $t_{r\rightarrow n}$). Thus, $\tau_{0}$ stays in its rhythmic state during $[t_{n\rightarrow r},t_{r\rightarrow n})$, and $t_{r\rightarrow n}=t_{n\rightarrow r}+\sum_{x=1}^{R}{P_{0,x}}$. Any packet of $\tau_{0}$ released in the system rhythmic mode is referred to as a rhythmic packet while the packets of task $\tau_{i}\;(1\leq i\leq n)$ are periodic packets. The delivery of packet $\chi_{i,k}$ at the $h$-th hop is referred to as a transmission denoted as $\chi_{i,k}(h)\ (1\leq h\leq H_{i})$.

Traditionally, RTWNs employ Link-based Scheduling (LBS) to allocate time slots for individual tasks where each slot is allocated to a link by specifying the sender and receiver [34]. If packets from different tasks share a common link and are both buffered at the same sender, their transmission order is decided by a node-specified policy (e.g., FIFO). This approach introduces uncertainty in packet scheduling and may violate the end-to-end (e2e) timing constraints on packet delivery. To tackle this problem, Transmission-based Scheduling (TBS) and Packet-based Scheduling (PBS) are proposed in [23] and [32], respectively, to construct deterministic schedules. Each of the two scheduling models has its own advantages and disadvantages and is preferred in different usage scenarios as discussed in [32]. Hence, we consider both models in our FD-PaS framework.

In the TBS model, each time slot is allocated to the transmission of a specific packet $\chi_{i,k}$ at a particular hop $h$ or kept idle. Once the network schedule is constructed, packet transmission in each time slot is unique and fixed. In the PBS model, each time slot is allocated to a specific packet $\chi_{i,k}$ or kept idle. Within each time slot assigned to $\chi_{i,k}$, every node along $\chi_{i,k}$’s routing path decides the action to take (e.g., transmit, receive or idle), depending on whether the node has received $\chi_{i,k}$ or not. For example, consider a task $\tau_{0}$ with two slots being assigned in each period. In the TBS model, the first and second slots are dedicated for $\tau_{0}$’s first and second hops, respectively. In the PBS model, the two slots are allocated to each packet of $\tau_{0}$ and the second slot can be used to transmit $\tau_{0}$’s first hop if the transmission fails in the first slot.

Since each link $e$ in the network may suffer packet losses, i.e., $\lambda^{L}_{e}<1$, packet transmissions may fail, which can significantly affect the timely delivery of real-time packets. To handle such cases, a retransmission mechanism is commonly employed in RTWNs [28, 30]. Specifically, if a sender node does not receive any ACK from the receiver node within the current slot, it automatically retransmits the packet in the next possible time slot.

To quantify the reliability requirement of the e2e packet delivery for each task, a required e2e PDR for $\tau_{i}$, denoted as $\lambda^{R}_{i}$, is introduced. Based on $\lambda^{R}_{i}$, the transmission of any packet of $\tau_{i}$ is reliable if and only if the achieved e2e PDR of $\tau_{i}$ is larger than or equal to $\lambda^{R}_{i}$, i.e., $\lambda_{i,k}\geq\lambda^{R}_{i}$. To simplify presentation, we assume that all tasks in the network share a common required e2e PDR value, denoted as $\lambda^{R}$. However, our proposed approach can be easily extended to support different $\lambda^{R}$’s for different tasks. Table II summarizes the frequently used symbols in this paper.

Based on the above system model, the problem that we aim to solve in this paper is presented as follows.

Problem 1: Assume that for a given RTWN, a static schedule is provided which can guarantee both the e2e timing and reliability requirements of all tasks when there are no disturbances. That is, required number of slots are assigned for each packet (either in the TBS model or PBS model) in the system nominal mode. Upon detection of a disturbance at $r_{0,m}$ (a release time of $\tau_{0}$’s packet333We assume that disturbances can be detected only at the time when the sensor samples the environment data, i.e., the release time of a certain packet.), determine the dynamic schedule in the system rhythmic mode such that (i) the system can start handling rhythmic packets no later than $r_{0,m+1}=r_{0,m}+P_{0}$, (ii) timing and reliability requirements of all the rhythmic packets are satisfied, and (iii) the system can safely return to the nominal mode after which all packets can be reliably delivered by their nominal deadlines. The objective is to minimize the total reliability degradation on all packets from periodic tasks in the system rhythmic mode.

Constraint (i) ensures that disturbances can be handled in the earliest possible time (i.e., before the nominal arrival time of the next packet). If Constraint (i) were violated, the corresponding control system could become unstable or suffer from severe performance degradation. The meaning of Constraints (ii) and (iii) are self explanatory.

It has been shown through a motivational example in [7] that centralized packet scheduling approaches (e.g. OLS and D2-PaS) have two main drawbacks when solving the above problem. First, they rely on a single point (e.g. the controller) in the network to make on-line decisions for handling the disturbance. This is a significant roadblock in scaling up the packet scheduling framework to be deployed in large-scale RTWNs. Secondly, centralized approaches suffer from a considerably long response time to the disturbances especially for large RTWNs. This is because centralized approaches require to first send the disturbance information to the controller. After that, a broadcast packet is needed to disseminate the generated dynamic schedule to all nodes in the network to handle the disturbance. In this work, we propose a new approach to address these drawbacks.

4 Overall Framework of FD-PaS

In order to achieve fast response to disturbances in RTWNs, in this work we propose a fully distributed packet scheduling framework, referred to as FD-PaS. The key idea of FD-PaS is to make dynamic, local schedule adaptation at each node along the path of the rhythmic task while avoiding transmission collisions from other nodes that still follow their static schedules in the system rhythmic mode.

Fig. 3 gives an overview of the execution model of FD-PaS. After network initialization, each node generates locally a static schedule, $S$, using the local schedule generation mechanism in D2-PaS and follows $S$ to transmit packets. When a disturbance is detected by rhythmic task $\tau_{0}$ at $t^{\prime}=r_{0,m}$, a notification is propagated to all the nodes responsible for handling the disturbance. Let these nodes be $V_{j}\in$ $\mathbf{V}_{rhy}$. Upon receiving the notification, each node in $\mathbf{V}_{rhy}$ determines the time duration of the network being in the rhythmic mode and generates a dynamic schedule $\tilde{S}$ for handling the disturbance. Starting from $r_{0,m+1}$, one nominal period of $\tau_{0}$ after detecting the disturbance, the nodes in $\mathbf{V}_{rhy}$ follow $\tilde{S}$ while all other nodes keep using static schedule $S$ to transmit periodic packets. Thus, by not relying on a broadcast packet to disseminate the dynamic schedule generated by a centralized point in the network, FD-PaS is able to significantly reduce the response time of reacting to disturbances. For ease of discussion, in the rest of the paper, we refer to disturbance response time (DRT) as the time duration from $t^{\prime}$ to the start time of system rhythmic mode and disturbance handling latency (DHL) as the time duration of system rhythmic mode (see Fig. 3).

To ensure that FD-PaS works properly, several challenges need to be tackled. First, when a disturbance occurs, only the sensor node that has detected it knows which task will enter the rhythmic state, while the rest of the nodes in $\mathbf{V}_{rhy}$ that are to handle the disturbance have no knowledge about this. Second, if the nodes in $\mathbf{V}_{rhy}$ follow the dynamic schedule while other nodes follow the static schedule $S$, transmission collisions would occur which may cause rhythmic packets to violate their timing and reliability requirements (e.g. missing deadlines). Third, to properly handle disturbances, efficient methods are needed by the nodes in $\mathbf{V}_{rhy}$ to determine a dynamic schedule in which the reliability degradation on periodic packets is minimized. We discuss in detail how FD-PaS tackles these challenges in the following sections.

5 Propagating Disturbance Information

In centralized approaches, all nodes in the RTWN must know the disturbance information since a dynamic schedule must be deployed at each node. However, such a network-wide propagation mechanism does not scale and often violates constraint (i) in Problem 1 as shown by the motivating example. To overcome this drawback, we propose to disseminate the disturbance information to only a subset of all nodes, denoted as $\mathbf{V}_{rhy}$, to minimize the DRT. This scheme requires the following three questions be answered: (1) which nodes in the network belong to $\mathbf{V}_{rhy}$, (2) how to propagate the disturbance information to nodes in $\mathbf{V}_{rhy}$, and (3) does each node in $\mathbf{V}_{rhy}$ have sufficient time to generate the dynamic schedule before the system enters the rhythmic mode? Below we present our answers to these questions.

Consider questions (1) and (2) above. Recall that when a disturbance occurs, the rhythmic task $\tau_{0}$ will enter its rhythmic state following reduced periods and deadlines as specified in $\overrightarrow{P_{0}}$ and $\overrightarrow{D_{0}}$. An updated schedule is needed to accommodate the increased workload of $\tau_{0}$. To ensure that each (re)transmission $\chi_{0,k}(h)$ can be successful, both the sender and the receiver of $\chi_{0,k}(h)$ must follow the same schedule. Thus, all nodes along the routing path of $\tau_{0}$ must know the disturbance information to generate a consistent dynamic schedule, and should be included in $\mathbf{V}_{rhy}$. For example, $\mathbf{V}_{rhy}$ $=\{V_{2},V_{c},V_{5}\}$ for the example in Fig. 1 when $\tau_{0}$ enters the rhythmic state. When a disturbance is detected at $r_{0,m}$, its information can be piggybacked onto $\chi_{0,m}$ and transmitted to all nodes in $\mathbf{V}_{rhy}$. Propagating disturbance information in this manner guarantees that all nodes in $\mathbf{V}_{rhy}$ receive the disturbance information within one nominal period of $\tau_{0}$, i.e., $P_{0}$, since the static schedule ensures that each task is assigned with the required number of transmission and retransmission slots along its routing path within $P_{0}$ in order to meet the e2e timing and reliability requirements.

Now consider question (3). As required in Constraint (i) of Problem 1, the system should start handling the rhythmic packets from $r_{0,m+1}$ after the disturbance is detected at $r_{0,m}$. This requires that (i) the disturbance information be successfully propagated to the relevant nodes before $\tau_{0}$ enters its rhythmic state at $r_{0,m+1}$, and (ii) each node in $\mathbf{V}_{rhy}$ completes the construction of the dynamic schedule before it starts receiving/transmitting the first rhythmic packet. The propagation scheme discussed above ensures that condition (i) is met. Regarding condition (ii), our prior work showed that one idle slot (10ms) is sufficient for a typical device node in RTWNs (e.g., TI CC2538 SoC) to complete its local schedule computation [23]. The theorem below establishes that such an idle slot indeed exists within the time frame specified in condition (ii).

Theorem 1**.**

*If an RTWN system is schedulable under a given static schedule, any node $V_{j}$ ($V_{j}\neq V_{c}$) in $\mathbf{V}_{rhy}$ has at least one idle slot (neither receiving nor sending any transmission) between time $t_{1}$ ($t_{1}\geq r_{0,m}$) when it receives the disturbance information and time $t_{2}$ ($t_{2}\geq r_{0,m+1}$) when it is involved in the transmission of the first rhythmic packet after $\tau_{0}$ enters its rhythmic state at $r_{0,m+1}$. *

Proof:

We first recall the following lemma from [23].

Lemma 1**.**

*If an RTWN system is schedulable under a given static schedule, i.e. each packet completes all its transmissions before the deadline, for any node $V_{j}\neq V_{c}$ and task $\tau_{i}$ passing through $V_{j}$, there exists at least one idle slot at $V_{j}$ among any three consecutive transmissions of $\tau_{i}$ passing $V_{j}$. *

Since in our system model, sensors and actuators are connected via the controller node, every task follows a routing path with at least two hops corresponding to two transmissions (assigned with multiple transmission and retransmission slots). Suppose $\chi_{0,m}(h)$ occurs at $t_{1}$ and is the transmission from which $V_{j}$ receives the disturbance information444If $V_{j}$ is the sensor, it detects the disturbance at $r_{0,m}$.. There exists at least one transmission between $\chi_{0,m}(h)$ and $\chi_{0,m+1}(h)$ (the first transmission that $V_{j}$ is involved in the dynamic schedule, occurring at $t_{2}$). Then, according to Lemma 1, $V_{j}$ has at least one idle slot between $\chi_{0,m}(h)$ and $\chi_{0,m+1}(h)$ (i.e., between $t_{1}$ and $t_{2}$). Thus, the theorem holds.

Based on Theorem 1 and the disturbance propagation time bound, the proposed partial disturbance propagation scheme guarantees that any disturbance can be promptly responded within one nominal period of the rhythmic task and Constraint (i) in Problem 1 can be satisfied.

6 Avoiding Transmission Collisions

According to the disturbance propagation mechanism presented in Section 5, only the nodes on the path of the rhythmic task are included in $\mathbf{V}_{rhy}$. Nodes in $\mathbf{V}_{rhy}$ construct their local schedules individually and employ them in the system rhythmic mode. All other nodes in the network follow the static schedule. With this execution model, unless the disturbance information is propagated to the entire RTWN, inconsistencies between the dynamic and static schedules in the system rhythmic mode may easily arise, which would result in transmission collisions. To ensure that the disturbances are handled appropriately, in the FD-PaS framework, the transmissions of rhythmic packets need to be always successful even in the presence of collision with other periodic packets.

In conventional RTWNs such as WirelessHART [28] and 6TiSCH [30], TDMA-based data link layer are widely adopted to provide synchronized and collision-free channel access. In addition, most of those protocols employ the Clear Channel Assessment (CCA) operation at the beginning of each transmission for collision avoidance. CCA, however, cannot prioritize packet transmissions. When multiple transmissions happen in the same time slot sharing the same destination, it cannot guarantee the more important packets (e.g., rhythmic packets) are granted the access to the channel.

To tackle this challenge, we propose an enhancement to the IEEE 802.15.4e standard [34], called Multi-Priority MAC (MP-MAC), to support prioritization of packet transmissions in RTWNs. Several attempts have been made in the literature towards supporting this feature. For example, the PriorityMAC was proposed in [27] to prioritize critical traffic in RTWNs. It introduces the concept of subslots, in which the transmitter does a very short transmission to indicate the priority of the packet to be transmitted in the following time slot. By adding two subslots before each time slot, PriorityMAC is able to create three priority levels. Different from PriorityMAC, the design of the MP-MAC aims to be lightweight and scalable. In MP-MAC, the transmitter does not explicitly conduct a short transmission to indicate the priority. Instead it implicitly indicates the priority of the transmission by adjusting the Start-Of-Frame (SOF) time offset. Compared with PriorityMAC, MP-MAC is more energy efficient (by avoiding transmissions in the subslots), and able to support more priority levels.

Fig. 4 gives a comparison of the slot timing of 802.15.4e (top) and MP-MAC (bottom). In a 802.15.4e time slot, the sender transmits a packet and the receiver responds with an acknowledgement (ACK) if the packet is successfully received555No acknowledgement is provided for broadcast and multicast packets.. The packet transmission starts at TxOffset after the start of the time slot, while the ACK starts at TxAckDelay after the completion of the packet transmission. A long Guard Time (LGT) and a short Guard Time (SGT) are used by the receiver and sender respectively to tolerate clock drift and radio/CPU operation delays. With this standard design of 802.15.4e, if multiple senders transmit packets in the same time slot, they are not aware of the other transmissions, and thus will cause interference. The slot timing of MP-MAC is presented at the bottom of Fig. 4. In MP-MAC, instead of being set as a constant, TxOffset is varied to implicitly indicate the priority of the packet (shown as red dashed lines). A packet with a higher priority is associated with a shorter TxOffset to start the transmission earlier. In addition, a CCA operation will be performed before each transmission to ensure that there is no higher priority packet transmission present in the channel. This enhancement ensures that only the highest priority packet (with the shortest TxOffset) is transmitted, and all lower priority transmissions yield to it.

Similar to the guard times, the TxOffset values for different priorities need to be set sufficiently apart so that different senders and receivers have consensus on the priorities. In MP-MAC, we define PriorityTick as the difference between two consecutive TxOffsets. To support $k$ different priorities in MP-MAC, the length of the time slot, compared to the standard design, needs be extended by $(k-1)\times PriorityTick$. A longer PriorityTick can ensure successful packet prioritization, but either leads to longer SlotDuration and reduced network throughput, or smaller number of supported priorities if the size of the time slot is fixed. Since PriorityTick is a hardware-dependent parameter, we will elaborate the selection of PriorityTick in our testbed experiments and demonstrate the effectiveness of MP-MAC in Section 9.1.

7 System Rhythmic Mode

MP-MAC ensures that once the dynamic schedules are generated locally, the nodes in $\mathbf{V}_{rhy}$ can follow those schedules to handle the disturbance without transmission collisions with other nodes in the network. Since all the nodes in $\mathbf{V}_{rhy}$ receive the same disturbance information, the dynamic schedules generated locally at these nodes are all consistent. The construction of a dynamic schedule must guarantee that 1) all rhythmic packets meet their timing and reliability requirements, 2) the reliability degradation of periodic packets is minimized, and 3) the system can reuse the static schedule after the rhythmic mode ends and all packets can be reliably delivered by their nominal deadlines.

7.1 Problem Formulation

In FD-PaS, the network starts operation by following a static schedule which guarantees that all tasks meet their timing and reliability requirements if no disturbance occurs. The static schedule is generated at each node locally using the local schedule generation technique proposed in [23]. To satisfy the reliability requirement, the retransmission mechanism introduced in [32, 31] is employed for each task to achieve the desired PDR value, i,e,, $\lambda_{i,k}\geq\lambda^{R}$. In the following, we assume the network adopts the TBS model where additional time slots are assigned to individual transmissions. (The case is similar for the PBS model where slots are assigned to individual packets.) We denote the static schedule as $S=\{(t,i,h)\}$, where $t$ is the slot ID, $i$ is the task ID and $h$ is the hop index. For any given time slot $t$, we have $S[t]=(i,h)$ if $t$ is assigned to the $h$-th transmission of $\tau_{i}$. Otherwise, $S[t]=(-1,-1)$ to indicate an idle slot. Let $\overrightarrow{R}_{i,k}=[R_{i,k}[0],R_{i,k}[1],\dots,R_{i,k}[H_{i}-1]]$ be the retry vector of packet $\chi_{i,k}$ used in the static schedule in which $R_{i,j}[h]$ denotes the number of slots assigned to hop $h$ of $\chi_{i,k}$. We use $w_{i}^{+}$ to denote the number of slots assigned to $\tau_{i}$ (i.e., $\overrightarrow{R}_{i,k}$) in the static schedule which guarantees the e2e PDR value $\lambda_{i,k}$ to be larger than $\lambda^{R}$ in the system nominal mode.

As shown in Fig. 2, when a disturbance is detected at $r_{0,m}$, $\tau_{0}$ requires to enter its rhythmic state from the next release time $r_{0,m+1}$, i.e., $t_{n\rightarrow r}=r_{0,m+1}$. Then, the system enters the rhythmic mode with an increased workload induced by $\tau_{0}$. A dynamic schedule $\tilde{S}$ is thus needed before the system switches back to the nominal mode and reuses static schedule $S$. $\tilde{S}$ starts from $t_{n\rightarrow r}$ and ends at a carefully chosen end point $t_{ep}$ of the system rhythmic mode. To achieve guaranteed fast disturbance handling, we further define $t_{ep}^{u}$ as a user specified parameter which bounds the maximum allowed DHL, and is often application dependent. Though it is natural to use idle slots in $S[t_{n\rightarrow r},t_{ep})$ to accommodate the increased rhythmic workload, they are not always sufficient to guarantee the timing and reliability requirements of all rhythmic packets. In this case, some periodic transmissions have to be dropped. Since any node $V_{j}\notin$ $\mathbf{V}_{rhy}$ keeps following the static schedule $S$ to transmit periodic packets, periodic transmissions cannot be adjusted in the dynamic schedule666Some periodic tasks may share common nodes with $\tau_{0}$ on their routing paths, which indicates that the periodic transmissions at these nodes can be adjusted in the dynamic schedule. Due to page limit, we leave this discussion to our future work and focus on the case that all periodic transmissions should not be adjusted in the dynamic schedule.. Therefore if any periodic transmission $\chi_{i,k}(h)$ in $S$ is replaced by a rhythmic transmission in $\tilde{S}$, the number of elements in $\overrightarrow{R}_{i,k}$ is reduced such that the reliability of packet $\chi_{i,k}$ is degraded. If the remaining number of assigned slots (denoted as $w_{i,k}$) is less than $H_{i}$, the timing requirement of $\chi_{i,k}$ is also violated since at least $H_{i}$ slots are needed to guaratee the delivery of $\chi_{i,k}$. To capture the reliability degradation for periodic packet $\chi_{i,k}$, let $\delta_{i,k}$ represent the difference between the required PDR $\lambda^{R}$ and the updated PDR value $\lambda_{i,k}$ in the dynamic schedule, i.e., $\delta_{i,k}=\max\{0,\lambda^{R}-\lambda_{i,k}\}$. Note that the timing degradation of each packet can also be captured by $\delta_{i,k}$ where $\delta_{i,k}=\lambda^{R}$ if $\chi_{i,k}$ is dropped. Then, the question is which periodic transmissions should be replaced by rhythmic transmissions to generate dynamic schedule $\tilde{S}[t_{n\rightarrow r},t_{ep})$ such that (i) all rhythmic packets meet their timing and reliability requirements and (ii) the total reliability degradation of periodic packets is minimized.

Formally, to satisfy Constraints (ii), (iii) and (iv) in Problem 1, we aim to solve the following two subproblems.

Problem 1.1 – End Point Selection: Given task set $\mathcal{T}$, $t_{n\rightarrow r}$, $t_{ep}^{u}$ and static schedule $S$, this subproblem determines the end point $t_{ep}$ that satisfies the following two constraints.

Constraint 1**.**

$f_{0,m+R}\leq t_{ep}\leq t_{ep}^{u}$* *

Here, $f_{0,m+R}$ is the finish time of the last packet released in $\tau_{0}$’s rhythmic state. $f_{0,m+R}\leq t_{ep}$ ensures that the current rhythmic event can be completely handled before the system switches back to the nominal mode.

Constraint 2**.**

*The system can switch back to the nominal mode and reuse the static schedule from $t_{ep}$ and all packets after $t_{ep}$ can be reliably delivered by their nominal deadlines. *

Problem 1.2 – Dynamic Schedule Generation: this subproblem generates the dynamic schedule $\tilde{S}[t_{n\rightarrow r},t_{ep})$ such that the total reliability degradation of periodic packets is minimized and the following two constraints are satisfied.

Constraint 3**.**

*All rhythmic packets meet their timing and reliability requirements. *

Constraint 4**.**

*In the dynamic schedule $\tilde{S}[t_{n\rightarrow r},t_{ep})$, any periodic transmission slot $S[t]=(i,h)(1\leq i\leq n)$ can only either be replaced by a rhythmic transmission slot $S[t]=(0,h)$ or kept unchanged. *

Below we first discuss how FD-PaS solves the first problem.

7.2 End Point Selection

Determining the right end point for the dynamic schedule is vital since it impacts not only the DHL but also the number of dropped periodic packets. A concept similar to the end point is used by OLS and is referred to as switch point [22]. Since both OLS and FD-PaS require the system to reuse the static schedule after $t_{ep}$, to select the end point in FD-PaS, we borrow some ideas in OLS including aligning the actual release time of $\tau_{0}$ to its nominal one and reducing the number of end point candidates by only considering the actual release times of $\tau_{0}$.

FD-PaS and OLS have two key differences for end point selection. First, to satisfy Constraint 2, we need to determine which packets must be completed before the system reuses the static schedule at end point $t_{ep}$. Since OLS must obey a user-specified bound on the number of adjusted transmissions in dynamic schedule $\tilde{S}$, a transmission set containing all transmissions to be scheduled in $\tilde{S}[t_{n\rightarrow r},t_{ep})$ must be constructed. However, FD-PaS has no such requirement (due to its distributed nature), thus only needs to construct an active packet set containing all packets to be scheduled. Second, according to Constraint 4, transmissions of periodic packets must not be adjusted and can only be replaced by rhythmic transmissions in the dynamic schedule. Thus, for the active packet set, we only need to consider rhythmic packets to be scheduled by $\tilde{S}[t_{n\rightarrow r},t_{ep})$. These differences require modifications to the end point selection process, which are detailed below.

Let $\Psi(t_{ep})$ denote the active packet set containing all rhythmic packets to be scheduled within $[t_{n\rightarrow r},t_{ep})$ and $\Phi(t_{ep})$ denote the periodic packet set in which each periodic packet has at least one transmission slot in the static schedule $S[t_{n\rightarrow r},t_{ep})$. Naturally, any rhythmic packet with both release time and deadline in $[t_{n\rightarrow r},t_{ep})$ must be included in $\Psi(t_{ep})$. The question is how to treat the rhythmic packet released before $t_{ep}$ with a deadline after $t_{ep}$. As shown in Fig. 2, let $\chi_{0,q^{*}}$ be such a packet. To ensure the system can reuse the static schedule from $t_{ep}$, the actual release time of $\tau_{0}$ must be aligned to its nominal release time after $t_{ep}$. Same as OLS, we shorten the time interval between $r_{0,q^{*}}$ and $r_{0,q^{*}+1}$ by shifting $r_{0,q^{*}+1}$ backward to the closest nominal release time of $\tau_{0}$, denoted as $r_{0,p^{*}}$. The more challenging part is adjusting the deadline and execution time of $\chi_{0,q^{*}}$ since the assigned number of transmission slots may vary depending on which hop occurs after $t_{ep}$. We construct $\chi_{0,q^{*}}$ by adjusting its execution time and deadline according to the position of $t_{ep}$ by considering the following two cases.

Case 1: If $t_{ep}<r_{0,p^{*}}$, $d_{0,q^{*}}$ is adjusted to $t_{ep}$. Suppose the first transmission slot assigned to $\tau_{0}$ after $t_{ep}$ is at $t_{k_{0}}$ in the static schedule. If $t_{k_{0}}\geq r_{0,p^{*}}$, it indicates that $S[t_{k_{0}}]$ is the first assigned transmission slot for the first hop of $\chi_{0,p^{*}}$, i.e., $S[t_{k_{0}}]=(0,1)$. Then the execution time of $\chi_{0,q^{*}}$ is set to $H_{0}$. If $t_{k_{0}}<r_{0,p^{*}}$ and suppose $S[t_{k_{0}}]$ is the $k_{0}$-th transmission slot assigned for $\chi_{0,p^{*}-1}$, the execution time is set to $k_{0}-1$ correspondingly.

Case 2: If $t_{ep}\geq r_{0,p^{*}}$, suppose the first assigned transmission slot for the first hop of $\chi_{0,p^{*}}$ is at $t_{1}$ in the static schedule, i.e., $S[t_{1}]=(0,1)(r_{0,p^{*}}\leq t_{1}<d_{0,p^{*}})$. $d_{0,q^{*}}$ is adjusted to $\min(t_{ep},t_{1})$ to guarantee that the deadline of $\chi_{0,q^{*}}$ is smaller than or equal to the first transmission of $\chi_{0,q^{*}+1}$. Also the execution time of $\chi_{0,q^{*}}$ is set to be equal to $H_{0}$.

Given $t_{ep}^{u}$, any time slot within $[f_{0,m+R},t_{ep}^{u}]$ can be selected as end point $t_{ep}$. However, to avoid checking every time instant which is time consuming, we only need to consider the actual release times of $\tau_{0}$ within $[f_{0,m+R},t_{ep}^{u}]$ as end point candidates, denoted as $t_{ep}^{c}$777Such a space reduction scheme is safe and can be proved in a similar way as Lemma 2 in [22] which is thus omitted due to page limit.. That is,

[TABLE]

Then the dynamic schedule generation subproblem can be refined as follow.

Problem 1.2: Given the end point candidate $t_{ep}^{c}$, active packet set $\Psi(t_{ep}^{c})$, periodic packet set $\Phi(t_{ep}^{c})$ and static schedule $S[t_{n\rightarrow r},t_{ep}^{c})$, determine the dynamic schedule $\tilde{S}[t_{n\rightarrow r},t_{ep}^{c})$ in which the total reliability degradation of periodic packets is minimized, i.e.,

[TABLE]

and Constraint 3 and Constraint 4 are satisfied.

8 Dynamic Schedule Generation

In this section, we discuss how FD-PaS determines the dynamic schedule to solve Problem 1.2. For the sake of clarity, we first assume that all links in the network are reliable, i.e. $\forall e,\lambda^{L}_{e}=100\%$. We then generalize the network model to consider lossy wireless links and extend FD-PaS to satisfy both the timing and reliability requirements of all tasks in Section 8.2.

8.1 Reliable Network Setting

For RTWNs in which all links are reliable, $H_{i}$ time slots are required for each packet $\chi_{i,k}$ to guarantee its e2e delivery. If any of the $H_{i}$ transmission slots in the static schedule is replaced by a rhythmic transmission in the dynamic schedule, $\chi_{i,k}$ cannot be delivered and has to be dropped. Thus, the objective in Eq. (2) is reduced to minimize the number of dropped periodic packets. We use $\rho[t_{n\rightarrow r},t_{ep})$ to denote the dropped periodic packet set and in the following we illustrate that determining $\rho[t_{n\rightarrow r},t_{ep})$ is a non-trivial problem by the following Lemma.

Lemma 2**.**

*Given end point $t_{ep}$, an active packet set $\Psi(t_{ep})$ containing all rhythmic packets of $\tau_{0}$ to be scheduled and a static schedule $S[t_{n\rightarrow r},t_{ep})$, determining the dropped packet set $\rho[t_{n\rightarrow r},t_{ep})$ with the minimum number of dropped packets and satisfying both Constraint 3 and Constraint 4 is NP-hard. *

Proof:

We prove the lemma by reducing the set cover problem [35] to a special case of the packet dropping problem.

The set cover problem is defined as follows: Given a set of $n$ elements $X=\{x_{1},x_{2},\dots,x_{n}\}$ and a collection $C=\{C_{1},C_{2},\dots,C_{m}\}$ of $m$ nonempty subsets of $X$ where $\cup_{i=1}^{m}C_{i}=X$. The set cover problem is to identify a sub-collection $C_{s}\subseteq C$ whose union equals $X$ such that $|C_{s}|$ is minimized.

Given a set cover problem, we can construct a special case of the packet dropping problem in polynomial time as follows:

(1) Suppose that after utilizing the original transmission slots of $\tau_{0}$ and the idle slots in $S[t_{n\rightarrow r},t_{ep})$ to accommodate rhythmic transmissions in $\Psi(t_{ep})$, there still remain $n$ packets of $\tau_{0}$, denoted as $\{x_{1},x_{2},\dots,x_{n}\}$, to be scheduled. Each packet $x_{i}$ only needs one slot to transmit.

(2) In the static schedule $S[t_{n\rightarrow r},t_{ep})$, there are $m$ periodic packets, denoted as $\{C_{1},C_{2},\dots,C_{m}\}$. For each packet $C_{j}$, if there exists a transmission of $C_{j}$ falls into the time window of rhythmic packet $x_{i}$ (i.e., $[r_{x_{i}},d_{x_{i}})$), we have $x_{i}\in C_{j}$.

Thus, one can determine the minimum number of dropped packet set $\rho[t_{n\rightarrow r},t_{ep})$ that can accommodate all the rhythmic packets if and only if the smallest sub-collection $C_{s}$ whose union equals $X$ can be identified. The Lemma is proved.

After the dropped packet set is determined, the dynamic schedule can be obtained in linear time by assigning the transmissions of the rhythmic packets to the static schedule $S[t_{n\rightarrow r},t_{ep})$ using both idle slots and transmission slots of the dropped packets. Thus Lemma 2 readily leads to Theorem 2 and the proof is omitted.

Theorem 2**.**

*Generating a dynamic schedule with the minimum number of dropped packets in reliable RTWNs is NP-hard. *

Below we focus on solving the packet dropping problem. An ILP based formulation can be derived by associating each periodic packet with a binary variable indicating whether the packet should be dropped or not. The objective is to minimize the number of dropped packets subject to the constraint that the total number of transmission slots freed from the dropped packets should be sufficient to meet the demand of all the rhythmic transmissions in $[t_{n\rightarrow r},t_{ep})$.

We introduce the following notation:

• •

$E_{j}=[\epsilon_{j}^{1},\epsilon_{j}^{2},\dots,\epsilon_{j}^{n}]\;(1\leq j\leq m)$ denotes the transmission vector of periodic packet $\chi_{j}$ where each $\epsilon_{j}^{i}$ is the number of transmissions from $\chi_{j}$ in the static schedule that can be replaced by transmissions of rhythmic packet $\chi_{0,i}$ in the dynamic schedule. Specifically, transmission $\chi_{j}(h)$ of $\chi_{j}$ can be replaced by $\chi_{0,i}$ if $S[t]=(j,h)$ and $r_{0,i}\leq t<d_{0,i}$.

• •

$l_{j}$ denotes the dropping decision of periodic packet $\chi_{j}$. $l_{j}=1$ if $\chi_{j}$ is dropped. Otherwise, $l_{j}=0$.

• •

$A=[a_{1},a_{2},\dots,a_{n}]$ denotes the available slot vector where each $a_{i}$ represents the total number of idle slots and rhythmic transmission slots in the static schedule that can be used by rhythmic packet $\chi_{0,i}$.

To drop the minimum number of periodic packets to guarantee the timing requirements of all rhythmic packets, we have the following objective function in the ILP formulation:

[TABLE]

Since rhythmic transmissions are at the highest priority, the deadline of each rhythmic packet $\chi_{0,i}$ can be guaranteed only if at least $H_{0,i}$ time slots are reserved for $\chi_{0,i}$ in the dynamic schedule. Also, both idle slots and rhythmic transmission slots in the static schedule can be used to satisfy $\chi_{0,i}$’s transmission demand. Therefore, objective function (3) is subject to the following constraint.

[TABLE]

Given that the packet dropping algorithm is to be deployed on resource-constrained device nodes and the sizes of both the rhythmic packet set ($|\Psi|$) and periodic packet set ($|\Phi|$) can become large as the network grows, we propose a greedy heuristic to solve the packet dropping problem which is time- and space-efficient to be deployed in practical RTWNs. The key idea of the greedy heuristic is to drop the periodic packet which contributes the maximum number of slots to all rhythmic packets.

Alg. 1 describes how the greedy heuristic drops periodic packets. Given the static schedule $S[t_{n\rightarrow r},t_{ep})$, a periodic packet set $\Phi=\{\chi_{j}|1\leq j\leq m\}$ in which each $\chi_{j}$ maintains a transmission vector $E_{j}=[\epsilon_{j}^{1},\epsilon_{j}^{2},\dots,\epsilon_{j}^{n}]$ is constructed (Lines 1$-$2). Considering the idle slots and rhythmic transmission slots in $S[t_{n\rightarrow r},t_{ep})$, the greedy heuristic constructs a demand vector $[v_{1},v_{2},\dots,v_{n}]$ for all rhythmic packets in $\Psi(t_{ep})$ where $v_{i}$ captures the number of additional slots required by $\chi_{0,i}$ ($v_{i}=H_{0,i}-a_{i}$) (Line 3). If all elements in the demand vector equal [math], which means that the idle slots and rhythmic transmission slots in the static schedule are sufficient to accommodate all rhythmic packets in $\Psi(t_{ep})$, no packet needs to be dropped and an empty set is returned (Lines 4$-$6). Otherwise, the heuristic drops packets in a greedy fashion as follows. In each iteration, periodic packet $\chi_{max}$ with the maximum $\sum_{i=1}^{n}\epsilon_{j}^{i}$ in $\Phi$ is added into the dropped packet set and removed from $\Phi$ (Line 8$-$9). Then the algorithm updates $\Psi(t_{ep})$’s demand vector by subtracting $\epsilon_{max}^{i}$ for each $v_{i}$ (Lines 10$-$12). If all rhythmic packets are schedulable, i.e., each $v_{i}$ equals [math], after dropping $\chi_{max}$, the dropped packet set $\rho[t_{n\rightarrow r},t_{ep})$ is returned (Lines 13$-$15). Otherwise, the transmission vector of each periodic packet $\chi_{j}$ is updated according to the status of rhythmic packets (Line 16). Specifically, if rhythmic packet $\chi_{0,i}$ is already schedulable, i.e., $v_{i}=0$, $\epsilon_{j}^{i}$ is set to [math]. If $0<v_{i}<\epsilon_{j}^{i}$ which means dropping $\chi_{j}$ is redundant to schedule $\chi_{0,i}$, we have $\epsilon_{j}^{i}=v_{i}$. This process repeats until all rhythmic packets are schedulable and a dropped packet set is returned.

The time complexity of the packet dropping heuristic, Algorithm 1, is $O(n\cdot m)$ where $n$ and $m$ are the number of rhythmic and periodic packets in the dynamic schedule, respectively.

8.2 Unreliable Network Setting

In the discussion above, we have assumed that all links in the RTWN are reliable, i.e., $\forall e,\lambda^{L}_{e}=100\%$. With this assumption, both timing and reliability requirements of each task can be directly satisfied when $H_{i}$ transmission slots are allocated for each packet and no retransmission slot is needed. Although this assumption simplifies the algorithm design and analysis, it is not realistic in real-life settings considering the lossy nature of wireless links. Thus, in this subsection we consider unreliable links and extend FD-PaS to handle disturbance considering both timing and reliability requirements for each task.

For RTWNs containing unreliable links, a retransmission mechanism is required and each packet may be assigned multiple retransmission slots in the static schedule according to the link quality on the routing path. After the system enters the rhythmic mode, Alg. 1 can still be applied if we do not differentiate transmission and retransmission slots allocated for each packet. That is, if any assigned slot of a periodic packet is determined to be occupied by a rhythmic transmission in the dynamic schedule, all its associated transmissions and retransmissions along the routing path will be dropped as well. However, this causes the system performance, in terms of both timing and reliability, to drop significantly since some of the dropped periodic transmissions may be kept to deliver this periodic packet. Then, the challenge is to determine the dropped periodic transmission set, denoted as $\rho^{*}[t_{n\rightarrow r},t_{ep})$, which leads to the minimum reliability degradation on periodic tasks (i.e., solving the problem defined in Eq. (2)).

Apparently, the packet dropping problem in Sec. 8.1, where dropping any transmission leads the same reliability degradation $\lambda^{R}$, is a special case of the transmission dropping problem considering unreliable link. Thus, according to Lemma 2, the following theorem holds and the proof is omitted.

Theorem 3**.**

*Generating a dynamic schedule with the minimum reliability degradation, i.e. solving Problem 1.2, is NP-hard. *

Next we focus on solving the transmission dropping problem and propose another heuristic. Note that, a packet may still be delivered even if some retransmissions are replaced by rhythmic transmissions. Thus, instead of dropping the packet contributing the maximum number of slots in Alg. 1, the key idea of the heuristic is to drop the periodic transmission which results in the minimum reliability degradation at each iteration. In the following we first describe the calculation of the reliability degradation for each transmission.

Given the PDRs of all the links along the routing path of $\tau_{i}$ and the retry vector $\overrightarrow{R}_{i,k}$, the reliability value of $\chi_{i,k}$, $\lambda_{i,k}$, can be derived as:

[TABLE]

If a retransmission of $\chi_{i,k}$ at $h$-th hop is dropped, the updated reliability value can be readily computed using Eq. (5) by updating $R_{i,k}[h]$ in the retry vector. The reliability degradation, then, is the difference between the two PDR values.

Alg. 2 describes the generation of the dropped transmission set using the heuristic. In the initialization phase, the periodic packet set and the rhythmic demand vector are constructed (Lines 1 - 2), and in Lines 3 - 5 we check whether any periodic transmission needs to be dropped in the dynamic schedule. If so, we drop periodic transmissions in a greedy manner. At each iteration, we select the periodic transmission $\chi_{j}(min)$ with the minimum reliability degradation according to the discussion above (Lines 7). If any time slot of $\chi_{j}(min)$ falls into the time window of any rhythmic packet needing extra slot to transmit, it is added into the dropped transmission set and the rhythmic demand vector is updated correspondingly (Lines 8-11). Otherwise, $\chi_{j}(min)$ is kept and cannot be selected in the future. If all rhythmic packets are schedulable, the dropped transmission set $\rho^{*}[t_{n\rightarrow r},t_{ep})$ is returned.

The time complexity of the dropped transmission determination is $O(n\cdot m\cdot w^{+})$ where $n$ and $m$ are the numbers of rhythmic and periodic packets in the dynamic schedule, respectively. $w^{+}$ is the number of slots assigned to each periodic packet in the static schedule.

Finally, with the dropped packet (transmission) set being determined, each node in $\mathbf{V}_{rhy}$ can readily generate the dynamic schedule to solve Problem 1.2 which is summarized in Alg. 3. According to our testbed experiments in Sec. 9.1, all nodes have runtime less than 1ms (within one time slot of 10ms) to complete the dynamic schedule generation.

Note that the proposed FD-PaS framework can be readily modified to handle disturbances in networks that adopt the PBS model. The only difference appears at the selection of the periodic transmission with the minimum reliability degradation (Line 7 in Alg. 2). Since time slots are allocated to each individual packet instead of transmission in the PBS model, we select the periodic packet with the minimum reliability degradation if one of the assigned slots is replaced by a rhythmic transmission. For computing the reliability value of each packet in the PBS model, readers can refer to [32].

9 Performance Evaluation

In this section, we present key performance results from both testbed experiments and simulation studies to evaluate the performance of the FD-PaS framework in RTWNs. The testbed implementation is to validate the correctness of the proposed FD-PaS framework and to obtain overhead in real applications. Extensive simulations are for performance evaluation since they allow us to easily vary taskset and network specifications to study the trend. Below we first introduce the experiments from our testbed.

9.1 Testbed Implementation and Evaluation

Our testbed is based on OpenWSN stack [36], an open source implementation of the 6TiSCH protocol [37]. OpenWSN enables IPv6 network over the TSCH mode of IEEE 802.15.4e MAC layer. A typical OpenWSN network consists of an OpenWSN Root and several OpenWSN devices, as well as an optional OpenLBR (Open Low-Power Border Router) to connect to IPv6 Internet. It serves as a perfect platform to experiment our proposed FD-PaS framework on both the data link and application layers of the stack.

We implemented FD-PaS on our RTWN testbed to validate the correctness of the design and evaluate its effectiveness for ensuring prompt response to unexpected disturbances. The MP-MAC was implemented by enhancing the MAC layer of the OpenWSN stack and the dynamic schedule generation algorithm (using the same code as in the simulation) was implemented in the application layer. In the following, we first present the implementation of MP-MAC and its performance evaluation, and then validate the correctness of FD-PaS in a multi-task multi-hop RTWN.

As shown in Fig.5, our testbed consists of 7 wireless devices (TI CC2538 SoC + SmartRF evaluation board). One of them is configured as the root node (controller node) and the rest are device nodes to form a multi-hop RTWN. A CC2531 sniffer is used to capture the packet. A 8-Channel Logic Analyzer is used to record device activities by physical pins, in order to accurately measure the timing information among different devices. Fig. 6 shows the experiment setup for the measurement of application layer performance.

9.1.1 Implementation and Evaluation of MP-MAC

For fair comparison with PriorityMAC [27], we used the 10ms slot timing of 802.15.4e in the MP-MAC implementation. Since PriorityMAC adds two subslots (0.4ms each) before each time slot, we also extended the SlotDuration and LongGT of MP-MAC by 0.8ms each. Table III summarizes the slot timing of MP-MAC, and the Adjusted TxOffset is computed as follows:

[TABLE]

With a given extended SlotDuration, the number of priority levels that MP-MAC can support, denoted as $N$, is a function of PriorityTick. In our MP-MAC implementation, $N$ is computed by $N=\left\lfloor{\frac{0.8ms}{\text{\em PriorityTick}}}\right\rfloor+1$. Fig. 7(a) shows how $N$ changes when the PriorityTick varies from 30$\mu$s to 400$\mu$s with a step size of 30$\mu$s (the timer resolution in the OpenWSN stack). Compared to PriorityMAC which can only support 3 effective priority levels, MP-MAC can support up to 14 priority levels in theory by extending the time slot with the same amount (0.8ms). Fig. 7(a) also illustrates the bandwidth improvement, defined as $\frac{10.8ms}{10ms+2\times PriorityTick}\times 100\%$, when MP-MAC only needs to maintain 3 priority levels. It can be seen that the bandwidth is improved by $7\%$ due to the reduction of the PriorityTick from $400\mu$s to $30\mu$s with 3 priority levels.

Measurement of Packet Error Rate (PER): Reducing the size of PriorityTick can support a larger number of priority levels in the RTWN system. Setting the PriorityTick too small, however, either causes nodes to lose synchronization, or make low priority senders unable to detect high priority packet transmissions and cause transmission collisions. It is thus important to identify safe PriorityTick values to make MP-MAC work appropriately. For this purpose, we set up a testing network with two senders talking directly to one receiver. We intentionally configure the senders to transmit in the same time slot and assign them with different priorities (using $D$ to denote the distance between the priority levels), and measure the number of correctly received packets on the receiver side. We define Packet Error Rate (PER) as the number of the failed transmissions divided by the number of total transmissions. During the test, each sender generates 10,000 packets. Fig. 7(b) shows the PER of the high priority packets by varying the size of PriorityTick from $400\mu$s to $30\mu$s. The PER of the low priority packets are always 100%, and are thus omitted in the figure. It can be observed that MP-MAC works properly under most of the PriorityTick settings. Its PER only increases when the PriorityTick is reduced to $30\mu$s. This indicates that the MP-MAC implementation on our device node (TI CC2538 SoC) can safely support up to 9 priority levels when the PriorityTick is set to be no less than $60\mu$s. When the PriorityTick is set at $30\mu$s, it also can be observed from Fig. 7(b) that the PER will drop (from around 10% to 5%) when the distance between the two priority levels increases (from $D=1$ to $D=2$).

Measurement of Application Layer Performance: To see how MP-MAC behaves in terms of packet transmission latency and packet drop rate (PDR) for different priority levels, we set up a testing network with three senders and a controller node. The three senders are assigned with different priorities (high, medium and low). Their schedules are configured in a way that they transmit in the same time slot every slotframe (with a length of $165m$s). The retransmission mechanism is enabled on all the senders so that if collision happens, the failed transmission retries in the next slotframe until a maximum number of $5$ retries is reached, and the packet is then dropped. We define packet drop rate (PDR) as the number of dropped packets divided by the number of total packets. We connect the controller node to a STM32F103 MCU through a UART port to control the packet generation on the senders. This STM32F103 MCU connects to the GPIO of each sender, and uses a pulse signal to trigger the sender to generate a packet. In the experiments, the controller node initiates and timestamps the packet generation. By comparing it to the timestamp of the packet reception, the application layer latency is obtained. After a successful packet reception, the controller node waits for a randomly selected time interval, and then triggers the next packet generation. To test latency and packet drop rate, we gradually reduce this time interval to increase the traffic volume. This will cause more transmission collisions in the network, which leads to more packet retransmissions and packet drop.

Fig. 8(a) and (b) show the PDR and application layer latency respectively for the three senders during the test. From the results, we observe that the packets from the high priority sender can always be transmitted in its first attempt while the medium and low priority senders have to yield upon collision by retransmission in future slotframes and suffer longer application layer latency. Similarly, when collision happens with the packets from the medium priority sender, the low priority sender has to yield again thus it is observed to have the longest latency. In Fig. 8(a), we note that the high priority packets can always guarantee the delivery and thus its PDR is consistently 0. On the other hand, both the low priority sender and medium priority sender experience increasing packet losses when the volume of the network traffic grows, and the impact on the low priority sender is more severe.

9.1.2 Functional validation in a multi-task multi-hop RTWN

We validate the correctness and effectiveness of FD-PaS by deploying it on a 7-node multi-hop network as shown in Fig. 1. The system running in the network consists of three tasks, $\tau_{0}=\{\{V_{0},V_{1},V_{c},V_{3},V_{4}\},15,8\},\tau_{1}=\{\{V_{2},V_{c},V_{3}\},30,6\}$ and $\tau_{2}=\{\{V_{1},V_{c},V_{5}\},20,4\}$. For each task, the first element denotes the routing path and the second one denotes its period (relative deadline). The third element represents the number of slots assigned to $\tau_{i}$, i.e. $w^{+}_{i}$, in the static schedule. We further assume that $\tau_{0}$ is the rhythmic task and $\overrightarrow{P_{0}}(\overrightarrow{D_{0}})=[12,12,12,12,12]$. The system starts running in the nominal mode at slot 1 and then switches to the rhythmic mode from slot 61. We use a Logic Analyzer to capture the radio activities from a pin of each device during time slot 1 - 120. In order to validate the effectiveness of FD-PaS in both reliable and lossy RTWNs, we deploy the heuristic presented in Alg. 1 to determine the dropped packet set and use the heuristic in Alg. 2 to determine the dropped transmission set, respectively. For the sake of clarity, we denote the latter as rFD-PaS. 888Both TBS and PBS models are tested. For simplicity, only the result from TBS is illustrated.

The captured results on our testbed are illustrated in Fig. 9. Specifically, Fig. 9a summarizes the legends. Fig. 9b shows the system nominal mode during time slot 1-60. Fig. 9c and Fig. 9d demonstrate the system rhythmic modes using FD-PaS and rFD-PaS during time slot 61-120, respectively.

In Fig. 9b, 9c, and 9d, 7 waveforms represent the radio activities (transmitting, receiving, or listening) for all the 7 nodes, as labeled on the left side of the figures. Each falling or rising edge of the waveform in the Slot row (lower part of the figures) marks the start of a new slot. In the bottom Schedule row, slot assignments are indicated using different colors and patterns. Each colored small block indicates the release time of the corresponding task at a certain node. Each transmission is denoted by a colored arrow of which the starting and ending points represent the sending and receiving nodes, respectively. In the rhythmic mode, a colored circle denotes a dropped periodic transmission preempted by a rhythmic one. For example, in Fig. 9b, $\tau_{1}$ releases its first packet at slot $1$ and is transmitted from $V_{2}$ to $V_{c}$ at slot 15.

Fig. 9b illustrates radio activities of the system in the nominal mode (1-60 slots), after which the system switches to the rhythmic mode. Given by the static schedule, each packet $\chi_{i,k}$ is allocated with extra slots for retransmission in the system nominal mode. But according to our testbed result shown in Fig. 9b, each transmission successes in its first assigned time slot without using any retransmission slot. During the rhythmic mode (slot 61-120), task $\tau_{0}$ releases 5 packets as indicated in Fig. 9c and Fig. 9d. To accommodate the increased workload in the system rhythmic mode, FD-PaS determines to drop both two packets of $\tau_{1}$ released in the system rhythmic mode. Since the sender of $\tau_{1}$, i.e. $V_{2}$, does not receive the disturbance information, it still follows the static schedule to transmit $\tau_{1}$ at the assigned slots (e.g. 75). However, to ensure the transmission of the rhythmic packets, all these periodic transmissions are preempted under our designed MP-MAC mechanism (indicated by circles). By contrast, rFD-PaS chooses not to completely drop two packets of $\tau_{1}$ but to reduce the number of slots assigned to $\tau_{1}$’s packets both from 6 to 4. In this case, both packets of $\tau_{1}$ still have chances to be successfully transmitted to the destination as illustrated in Fig. 9d. This significantly increases the reliability of $\tau_{1}$ compared to that under the dropping decision made by FD-PaS. These results above match those from the simulation of FD-PaS and rFD-PaS under the same experiment settings.

9.2 Simulation Studies

9.2.1 Simulation Setup

In the simulation studies, we compare FD-PaS with both OLS and D2-PaS approaches that are able to handle unexpected external disturbances in RTWNs. The following two key performance metrics are used in the studies.

Success Ratio (SR): SR is defined as the fraction of feasible task sets over all the generated task sets. A task set is feasible only if a specified DRT can be achieved.

Degradation Rate (DR): DR is defined as the ratio between the sum of reliability degradation from all periodic packets (i.e. $\sum\delta_{i,k}$) and the total number of generated periodic packets in the system rhythmic mode.

For fair comparison, we use randomly generated task sets. Each random task set is generated according to a target nominal utilization $U^{*}$ and by incrementally adding random periodic tasks to an initially empty set $\mathcal{T}$. The generation of each random task $\tau_{i}$ is controlled by the following parameters: (i) the number of hops $H_{i}$ drawn from the uniform distribution over $\{2,3,\dots,16\}$, (ii) nominal period $P_{i}$ drawn from the uniform distribution over $\{H_{i},\dots,500\}$, and (iii) nominal relative deadline $D_{i}$ equal to period $P_{i}$999 The unit of $P_{i}$ and $D_{i}$ values is one time slot and the range of the parameters are determined according to realistic RTWN applications.. To guarantee reliable transmission, we use the TBS model to determine the number of slots assigned for each packet according to Eq. (5).

After all tasks in $\mathcal{T}$ are generated, we randomly select one of them as the rhythmic task $\tau_{0}$ and assume that the disturbance is detected at the $k$-th instance of $\tau_{0}$ where $k$ is randomly selected from $\{1,\dots,20\}$. The period vector $\overrightarrow{P_{0}}$ ($\overrightarrow{D_{0}}=\overrightarrow{P_{0}}$) is generated by controlling the following parameters: (i) the number of elements in $\overrightarrow{P_{0}}$, $R$, and (ii) the initial rhythmic period ratio, $\gamma=P_{0,1}/P_{0}$. To better control the workload of the rhythmic task, we fix $\gamma$ to $0.2$ and tune $R$ which can be any integer in the set of $\{4,6,\dots,16\}$. Given $\gamma$ and $R$, the value of each rhythmic period $P_{0,k}$ can be computed by $P_{0,k}\,(1\leq k\leq R)=\lfloor P_{0}\times(\gamma+(k-1)\times\frac{1-\gamma}{R})\rfloor$.

Additional parameters needed are summarized as follows: 1) the maximum allowed DRT $\alpha$ which is some integer multiple of the nominal period of the rhythmic task $P_{0}$; 2) the end point scaling factor $\beta$ which determines the upper bound of the end point $t_{ep}^{u}$ where $t_{ep}^{u}=t_{r\rightarrow n}+(\beta-1)\times P_{0}$. Naturally, a larger $\beta$ will lead to better performance in terms of a lower reliability degradation but may cause longer DHL. To keep $\beta$ as small as possible without performance degradation, we set $\beta=4$ which means the disturbance must be completely handled within 3 nominal periods after the rhythmic task returns to its nominal state. Other parameters used in OLS and D2-PaS, e.g. the payload size of a broadcast packet, are set to the same as that in [23] for fair comparison.

9.2.2 Simulation Results

In the first set of experiments, we compare the SR of OLS, D2-PaS and FD-PaS for randomly generated task sets with a nominal utilization $U^{*}=0.5$ (see Fig. 10) by varying the maximum allowed DRT, $\alpha$, from $P_{0}$ to 6$P_{0}$, with a step size of $P_{0}$. The results with other target nominal utilization show similar behavior and thus are omitted. In the experiments, each data point is based on 10,000 randomly generated task sets. As can be observed from Fig. 10, D2-PaS and OLS have exactly the same SR because they both rely on a broadcast packet to propagate the disturbance information to the entire network. Under D2-PaS and OLS, the task sets are all feasible only when $\alpha=6P_{0}$, i.e., the maximum allowed DRT is set to be $6$ nominal periods of the rhythmic task. However, in most practical settings, the RTWN is required to provide fast response to the disturbance within one nominal period, i.e., $\alpha=P_{0}$. In this case, the SR of both D2-PaS and OLS drops to $25\%$. On the other hand, FD-PaS can always achieve $100\%$ SR since the RTWN can start handling disturbance from the beginning of the next nominal period as required by Constraint (i) of Problem 1 in FD-PaS.

In the second set of experiments, we compare the average DR of OLS, D2-PaS and FD-PaS by varying the nominal utilization $U^{*}$ of the randomly generated task sets and the number of rhythmic periods $R$ of the selected rhythmic task. As both OLS and D2-PaS do not consider unreliable links in packet scheduling, we first extend them to support reliable transmission. Specifically, all packets in OLS and D2-PaS are reliably transmitted using $w_{i}^{+}$ slots in the static schedule. In the dynamic schedule, transmission and retransmission slots assigned to each packet are not differentiated, i.e., each packet can either be reliably scheduled or dropped. For our proposed framework, both FD-PaS and rFD-PaS are simulated to determine the dropped packet and transmission sets in reliable and lossy RTWNs, respectively.

Fig. 11 and Fig. 12 summarize the average DR as a function of $U^{*}$ and $R$, respectively, where each data point is the average value of $1,000$ trials. From the figures, we can observe that both FD-PaS and rFD-PaS have significantly lower average DR over OLS ($53\%$ on average and $82\%$ in the best case). The much higher DR of OLS is due to OLS’s large broadcast overhead resulted from the centralized approach. Compared to D2-PaS, FD-PaS drops around $12\%$ more periodic packets on average since FD-PaS makes local packet dropping decisions thus tends to drop more packets. On the other hand, rFD-PaS has slightly higher average DR over D2-PaS ($1.4\%$ on average), which is contrary to our expectation since rFD-PaS has higher flexibility on adjusting the dynamic schedule. The main reason here is that the FD-PaS framework suffers from the possibility of infeasible end point selection since the system must reuse the static schedule after a well-selected end point. Nonetheless, with the significant improvement in the average success ratio ($75\%$ when the maximum allowed DRT is set to be $P_{0}$), the degradation in DR is acceptable.

Another observation from Fig. 12 is that the average DR of FD-PaS first drops when $R$ increases from $4$ to $10$, and then increases when $R$ keeps increasing from $10$ to $16$. One would expect that the average DR should monotonically increase when both $U^{*}$ and $R$ increase (as the case for both OLS and D2-PaS). Through extensive simulation studies (detailed results are omitted due to page limit), we observe that the average DR of FD-PaS is highly dependent on slot utilization (fraction of number of slots contributed by the dropped periodic packets, used by rhythmic packets). When $R$ increases from a small value (e.g., 4), the slot utilization also increases101010For example, when $R$ is small, even if the rhythmic packet only needs one slot from a periodic packet, the whole periodic packet has to be dropped and all its other assigned slots are wasted.. That is, though we need to drop more packets when the workload of the rhythmic task increases, the number of dropped packets grows more slowly than the number of packets in the system rhythmic mode. This explains why the DR of FD-PaS decreases when $R$ increases from $4$ to $10$. On the other hand, we found that when $R$ keeps increasing, the slot utilization starts decreasing, and the number of dropped packets grows faster than the total number of packets in the system rhythmic mode. This leads to the observation that the DR of FD-PaS starts to increase from $10$ to $16$.

10 Conclusion and Future Work

In this paper, we propose FD-PaS, a fully distributed packet scheduling framework, to handle unexpected disturbances in lossy RTWNs. Unlike centralized approaches where dynamic schedules are generated in the controller node and disseminated to the entire network, FD-PaS makes on-line decisions to handle disturbances locally without any centralized control. Such a fully distributed framework not only significantly improves the scalability but also provides guaranteed fast response to external disturbances. Our FD-PaS framework including both the multi-priority data link layer design and the dynamic schedule construction method is implemented on our RTWN testbed. Extensive experiments have been conducted to validate its correctness and effectiveness. As future work, we will extend FD-PaS to support multi-channel settings and will explore how to handle concurrent disturbances in a fully distributed manner.