RT-Gang: Real-Time Gang Scheduling Framework for Safety-Critical Systems

TL;DR

RT-Gang is a new real-time scheduling framework that enforces one gang at a time to improve predictability in multicore systems, using memory bandwidth throttling to safely run best-effort tasks.

Contribution

It introduces a one-gang-at-a-time scheduling policy with integrated memory bandwidth throttling, enhancing predictability for safety-critical multicore systems.

Findings

Significantly improves system predictability.

Negligible overhead in implementation.

Effective in real-world DNN workloads.

Abstract

In this paper, we present RT-Gang: a novel real-time gang scheduling framework that enforces a one-gang-at-a-time policy. We find that, in a multicore platform, co-scheduling multiple parallel real-time tasks would require highly pessimistic worst-case execution time (WCET) and schedulability analysis - even when there are enough cores - due to contention in shared hardware resources such as cache and DRAM controller. In RT-Gang, all threads of a parallel real-time task form a real-time gang and the scheduler globally enforces the one-gang-at-a-time scheduling policy to guarantee tight and accurate task WCET. To minimize under-utilization, we integrate a state-of-the-art memory bandwidth throttling framework to allow safe execution of best-effort tasks. Specifically, any idle cores, if exist, are used to schedule best-effort tasks but their maximum memory bandwidth usages are strictly throttled to tightly bound interference to real-time gang tasks. We implement RT-Gang in the Linux kernel and evaluate it on two representative embedded multicore platforms using both synthetic and real-world DNN workloads. The results show that RT-Gang dramatically improves system predictability and the overhead is negligible.

Tables

Table 1. TABLE I: Taskset parameters of the illustrative example

Task	WCET ($C$)	Period ($P$)	# Thread
${\tau }_1^{RT}$	2	10	2
${\tau }_2^{RT}$	4	10	2
${\tau }_3^{BE}$	$\infty$	N/A	4

Table 2. TABLE II: Taskset parameters for the DNN experiment.

Task	WCET ($C$ ms)	Period ($P$ ms)	# Threads
Common
${\tau }_{cutcp}^{BE}$	$\infty$	N/A	4
${\tau }_{lbm}^{BE}$	$\infty$	N/A	4
Jetson TX2
${\tau }_{bww}^{RT}$	40.0	100.0	4
${\tau }_{dnn\left(2\right)}^{RT}$	10.7	24.0	2
${\tau }_{dnn\left(3\right)}^{RT}$	8.8	19.0	3
${\tau }_{dnn\left(4\right)}^{RT}$	7.6	17.0	4
Raspberry Pi 3
${\tau }_{bww}^{RT}$	47.0	100.0	4
${\tau }_{dnn\left(2\right)}^{RT}$	34.0	78.0	2
${\tau }_{dnn\left(3\right)}^{RT}$	27.90	65.0	3
${\tau }_{dnn\left(4\right)}^{RT}$	24.81	56.0	4

Table 3. TABLE III: RT-Gang Overhead in Linux

Scenario	Context Switch Cost (usec)
1-Thread-Lowprio (Linux)	6.81
1-Thread-Lowprio (RT-Gang)	7.19
2-Thread-Lowprio (RT-Gang)	7.37
3-Thread-Lowprio (RT-Gang)	7.55
4-Thread-Lowprio (RT-Gang)	7.72

Full text

In RT-Gang, all threads of a parallel real-time task form a real-time gang and the scheduler globally enforces the one-gang-at-a-time scheduling policy to guarantee tight and accurate task WCET. To minimize under-utilization, we integrate a state-of-the-art memory bandwidth throttling framework to allow safe execution of best-effort tasks. Specifically, any idle cores, if exist, are used to schedule best-effort tasks but their maximum memory bandwidth usages are strictly throttled to tightly bound interference to real-time gang tasks.

We implement RT-Gang in the Linux kernel and evaluate it on two representative embedded multicore platforms using both synthetic and real-world DNN workloads. The results show that RT-Gang dramatically improves system predictability and the overhead is negligible.

I Introduction

Safety-critical embedded real-time systems in automotive and aviation industries increasingly demand efficient, high-performance computing platforms to execute compute and data intensive workloads (e.g., deep neural networks) in real-time, while meeting size, weight, power and cost constraints  [29]. However, engineers and researchers developing and studying these safety-critical systems have had troubles to deal with modern high-performance computer architectures because of their unpredictable and extremely poor worst-case timing behaviors that are too complex to understand and analyze [5].

In a safety-critical real-time system, the use of high-performance multicore platforms is challenging because shared hardware resources, such as cache and memory controllers, can cause extremely high timing variations [57, 51]. The timing unpredictability is a serious problem in both automotive and aviation industries. For example, Bosch, a major automotive supplier, recently announced “predictability on high-performance platforms” as a major industrial challenge for which the industry is actively seeking solutions from the research community [20]. In aviation, the problem was dubbed as “one-out-of-m” problem [25] because the current industry practice is to disable all but one core as recommended by the Federal Aviation Administration (FAA) for certification, which requires evidence of bounded interference [12].

Prior efforts to address the timing predictability problem have been largely based on the two basic ideas: (1) designing simpler time-predictable architectures and (2) partitioning shared resources. Unfortunately, simpler architectures tend to trade-off too much performance in favor of predictability, which we can no longer afford. Partitioning shared resources improves predictability but cannot guarantee tight worst-case timing in high-performance multicore architectures because there are too many important but unpartitionable shared resources [51]. Moreover, not only partitioning generally reduces efficiency and maximum achievable performance, but also it is very difficult to partition properly for parallel tasks, while many emerging real-time workloads, such as deep neural network (DNN) [9, 37], are highly parallel.

In this paper, we present RT-Gang: a novel real-time gang scheduling framework that can efficiently and predictably utilize modern high-performance multicore architectures for safety-critical real-time systems. We focus on supporting emerging parallel real-time workloads, such as DNN-based real-time sensing and control tasks [9, 37], while also supporting legacy single-threaded real-time applications. Our key observation is that, from the worst-case execution time (WCET) standpoint, scheduling one parallel real-time task at a time is better than co-scheduling multiple parallel real-time tasks, because the latter case must assume highly pessimistic WCETs on multicore architecture (see Section II).

In RT-Gang, all threads of a parallel real-time task form a real-time gang and the scheduler globally enforces a one-gang-at-a-time scheduling policy to guarantee tight and accurate task WCET. Assuming all real-time tasks are parallelized and each parallel task can fully take advantage of all the available computing resources on the platform, this one-gang-at-a-time approach essentially renders parallel real-time task scheduling on multicore into the well-understood single-core real-time scheduling problem [43]. The most significant benefit of this transformation is that we no longer need to worry about shared resource interference because all scheduled threads are part of a single real-time task and that resource sharing is constructive rather than destructive in this setting. Assuming the WCET of a parallel real-time task is estimated in isolation, RT-Gang guarantees that the WCET will be respected regardless of other tasks on the system. Furthermore, we can apply well-understood single-core based real-time task scheduling policies and analysis methodologies [28, 47, 4] without making strong assumptions on the underlying multicore architectures.

Assuming all real-time tasks are perfectly parallelized is, however, unrealistic. Many real-time tasks are difficult or impossible to parallelize. Also, parallelized code often does not scale well. Therefore, our one-gang-at-a-time policy can significantly under-utilize computing resources when imperfectly parallelized or single-threaded real-time tasks are scheduled one-at-a-time. We address the potential resource under-utilization problem in the following two ways.

First, we introduce the notion of a virtual gang. We define a virtual gang task as a group of real-time tasks that are explicitly linked and scheduled together as if they are threads of a single parallel real-time task under our gang scheduler. Although real-time tasks in a virtual gang could interfere with each other, because the members of a gang are statically determined at design time, analyzing their WCET impacts, while not the focus of this paper, is easier under RT-Gang (e.g., via measurement).

Second, we allow co-scheduling of best-effort tasks on any of the available idle system cores but with a condition that the cores are strictly regulated by a memory bandwidth throttling mechanism [55]. Each real-time gang defines its tolerable maximum memory bandwidth, which is then strictly enforced by the throttling mechanism to ensure bounded interference to the real-time gang task.

We implement RT-Gang in Linux kernel and evaluate it on two representative embedded multicore platforms, NVIDIA Jetson TX-2 and Raspberry Pi 3, using both synthetic and real-world workloads. The results show that RT-Gang dramatically improves system predictability while the measured overhead is negligible.

In summary, we make the following contributions:

• •

We propose RT-Gang, a novel gang scheduling framework that enables analyzable parallel real-time scheduling on multicore platforms.

• •

We introduce the virtual gang concept to minimize resource utilization by statically grouping multiple real-time tasks as a single schedulable real-time gang.

• •

We further improve efficiency by integrating a state-of-the-art memory bandwidth throttling framework to enable safe best-effort task execution on any existing idle cores.

• •

We implement RT-Gang in Linux kernel and present its evaluation results on two different embedded platforms using both synthetic and real-world workloads. We also provide RT-Gang as open-source [2].

The rest of this paper is organized as follows: We present a motivating case-study in Section II. In Section III, we present the design of RT-Gang, and in Section IV, we describe its implementation details in Linux. In Section V, we present our evaluation results. After discussing potential use-cases and related work in Section VI and Section VII, respectively, we conclude in Section VIII.

II Motivation

In this section, we provide evidence that shows why scheduling one real-time gang at a time can be better from the perspective of task WCETs through a case-study.

In this case-study, we use a deep neural network (DNN) based real-time control task of DeepPicar [7] as our workload. The control loop uses a single deep neural network (DNN) to produce the car’s steering angle control output from raw images of the car’s front-facing camera in real-time. Importantly, its DNN architecture is the same as the one used in NVIDIA’s real self-driving car called DAVE-2 [9].

Note that DNN processing is highly compute and data intensive, which is thus often parallelized to improve performance. Figure 1(a) shows the average execution times of the control loop while varying the number of CPU cores used on a quad-core embedded platform (Raspberry Pi 3). It can be seen that as we assign more cores for DNN processing, the performance improves—from 46.30 ms on a single core to 22.86 ms on four cores. If the real-time requirement is 30Hz, one might want to parallelize the DNN using two cores, while co-scheduling other tasks on the other two remaining cores.

Figure 1(b) shows the timing impact of such co-scheduling, where the DNN control task and a memory intensive task are scheduled first in isolation (Solo) and then together (Corun).

The interesting, and troubling, observation is that the two tasks experience dramatically different timing impact: the DNN control task suffers 10.33X slowdown, while the memory benchmark suffers only 1.05X slowdown.

For safety-critical real-time systems, this means that extremely pessimistic task WCETs must be assumed to be safe. The potential timing impact of interference highly depends on task’s memory access patterns and the underlying hardware. For example, we observed more than 100X slowdown (two orders of magnitudes) using a linked-list iteration task on the same computing platform used above, even after we partition core as well as the shared cache among the tasks. Similar degrees of timing impacts of interference have been reported in recent empirical studies on contemporary embedded multicore platforms [52, 51, 57, 7, 8].

When a task’s WCET has to be assumed 10X or 100X its solo execution time, we can see why in aviation industry, it makes perfect sense to disable all but one core [25] and why the certification authorities in US and Europe recommend it for certification [12, 11]. However, disabling cores obviously defeats the purpose of using multicore platforms in the first place—the need of more performance.

In our DNN control-task case-study above, a better approach is to use all four cores just for the parallelized DNN processing task—without co-scheduling—which would result in quicker completion of the control task. More importantly, because there would be no other competing co-scheduled tasks, there’s no need to pessimistically estimate the task’s WCET. This, in turn, will achieve better overall schedulability.

In this sense, from the WCET standpoint, scheduling fully parallelized tasks one-at-a-time might be better than co-scheduling multiple of them at the same time. Generally applying this approach, however, has two major issues. First, not all real-time tasks can be easily parallelized. Second, parallelization often does not scale due to synchronization and other overheads. Therefore, the danger is that some cores may be under-utilized under the one-at-a-time scheduling policy.

In summary, we have made a case why, from the WCET standpoint, scheduling one parallel real-time task at a time can be better than co-scheduling multiple parallel tasks simultaneously, although possible under-utilization of the computing resources is a concern.

III RT-Gang

In this section, we describe the design of RT-Gang.

III-A Assumptions and Objectives

We consider a shared memory based multicore processor. We assume a hierarchical OS scheduling framework in which real-time tasks are strictly prioritized over best-effort tasks (e.g., Linux). We assume that each task is composed of one or more threads. We assume that the thread to core assignment of each parallel real-time task, but not best-effort ones, is given and fixed (i.e., no migration). We assume that the WCET of each real-time task is either experimentally measured or analytically computed (e.g., [44]) in isolation.

Our objective is to eliminate the need to analyze interference between the real-time tasks on a multicore platform by turning multicore parallel real-time scheduling into an equivalent of single-core real-time task scheduling, while minimizing potential utilization loss.

III-B Design Overview

RT-Gang is based on a simple idea: schedule one real-time task—parallel or not—at a time. When a real-time task is released, all of its threads, called a gang, shall be scheduled all at once—if the task is the highest priority one—or not at all—if the task’s priority is lower than the currently scheduled real-time task—even if there exist some idle cores. In other words, we implement a version of gang scheduler, but unlike prior gang scheduler designs [19, 23, 17], we do not allow co-scheduling of other real-time tasks even when there are available idle cores. We do allow, however, co-scheduling best-effort tasks with strictly imposed limits on their allowed memory access rates by integrating an existing memory bandwidth throttling mechanism [55, 3].

In our approach, each real-time task declares its maximum allowed interfering memory traffic from the best-effort tasks on different cores, which is then enforced by the OS at runtime for the cores that schedule best-effort tasks, if such cores exist. In this way, the parallel real-time task, a real-time gang, is guaranteed to be able to use all available computing resources, and the maximum interference is strictly limited to a certain threshold value, determined by the task itself in advance. If the real-time gang needs maximum isolation, it can set its interference threshold value to be zero, preventing any co-scheduling of best-effort tasks.

Figure 2 shows an example schedule under RT-Gang framework. In this example, three real-time tasks $t_{1}$, $t_{2}$, and $t_{3}$ (in increasing priority) are scheduled. The task $t_{1}$ has four threads, while $t_{2}$ and $t_{3}$ have three threads and one thread, respectively.

At first, $t_{1}$ is scheduled. When $t_{2}$ is released, it preempts $t_{1}$ because it has higher priority. Note that even if Core 3 and 4 are idle at the time, $t_{1}$ cannot use the cores. When $t_{3}$, a single-threaded task, becomes ready, all cores except Core 3 become idle to guarantee that $t_{3}$ is the only real-time task in the entire system. In this way, our real-time gang scheduler strictly enforces the one real-time gang at a time policy.

Note that the dot-filled rectangles are slack-durations during which best-effort tasks can be scheduled, using a general purpose scheduler, such as Linux’s Completely Fair Scheduler (CFS) [35], but they will be throttled based on each real-time task’s declared tolerable threshold value.

RT-Gang’s design approach offers several major benefits. First, by scheduling one real-time gang at a time, we no longer need to worry about interference from other real-time tasks. Possible interference from best-effort tasks is strictly regulated by the threshold value determined by the task itself. Thus, we can obtain accurate WCET of a real-time task (e.g., measure the timing while injecting the threshold amount of memory traffic). Also, as shown in [39], we can obtain better analytic memory interference bounds when we control the amount of competing memory traffic. In other words, we no longer need to deal with highly pessimistic 10X or 100X WCETs. An equally desirable aspect of this approach is that it renders the problem of scheduling parallel real-time tasks on multicore as the simple, well-understood classical real-time scheduling problem on single-core systems [28, 47]. Thus, we can directly apply classical single-core analysis methods [4].

III-C Virtual Gang

Our basic real-time gang scheduling approach described above would work well when all real-time tasks are perfectly parallelized, but it may under-utilize the system if real-time tasks are single-threaded and best-effort tasks are not available. Furthermore, some of them may miss the deadlines, while they might be schedulable if all cores are used.

We propose the virtual gang concept to mitigate the resource under-utilization problem of imperfectly parallelized real-time tasks under our gang scheduling policy. We define a virtual gang as a group of real-time tasks that are explicitly linked and scheduled together as if they are threads of a single real-time task under our gang scheduler. A virtual gang task has a fixed priority value that applies to all of its members. From the gang scheduler’s point of view, a virtual gang is just like a normal real-time task, except the scheduled threads are not from the same task but from multiple different tasks. The composition of a virtual gang must be explicitly determined by the system designer at design time. Once a real-time task becomes a member of a virtual gang, all of its threads are scheduled simultaneously with the threads of the other member real-time tasks of the virtual gang.

Figure 3 shows two example schedules involving a virtual gang $t_{g}$, which consists of three separate real-time tasks: $t_{1},t_{2}$ and $t_{3}$. Note that $t_{1}$ and $t_{2}$ are single-threaded while $t_{3}$ is multi-threaded but only two threads are used. All real-time tasks in the virtual gang $t_{g}$ share the same priority. Therefore, from the scheduler’s point-of-view, all threads of the three tasks are treated as a single (virtual) real-time task $t_{g}$, the virtual gang task. For instance, in inset (a) of Figure 3, a newly arrived task $t_{4}$ cannot be scheduled until $t_{1}$’s last thread is completed because $t_{g}$’s priority is higher than $t_{4}$. In inset (b), on the other hand, $t_{4}$ immediately preempts all active threads of the virtual gang $t_{g}$, because $t_{g}$’s priority is lower than $t_{4}$’s.

Although the real-time tasks in a virtual gang can destructively interfere with each other, their membership is explicitly determined at the design time. Therefore, the effect of shared resource interference among the tasks in a virtual gang can be carefully analyzed, either empirically or analytically, and taken into account for system-level schedulability analysis.

III-D Safe Best-Effort Task Co-Scheduling

Because our real-time gang scheduling approach strictly disallows concurrent real-time tasks, which are not part of the currently scheduled real-time gang, it is possible that some cores may be idle. As we discussed earlier, we allow scheduling of best-effort tasks on those idle cores with a condition that their interference is strictly bounded by integrating a memory bandwidth throttling mechanism as found in [55].

The throttling mechanism in  [55] uses per-core hardware performance counters to bound the maximum number of memory transactions within a given time interval (e.g., 1-msec period) to a certain programmable threshold (budget). When the core reaches the programmed threshold, the hardware counter generates an overflow interrupt, which is then handled by the OS to stop the core until the next time period begins.

Assuming that the currently scheduled real-time gang is actively using $k$ cores out of $m$ cores ($k\leq m$), there are $m-k$ idle cores on which we can schedule best-effort tasks—i.e., those that do not have hard real-time requirements. The best-effort tasks scheduled on the idle cores are given strict limits in terms of their memory bandwidth usage so that their impact to the real-time gang is bounded. The bandwidth limit of the best-effort tasks is determined by the currently scheduled real-time gang in the system. When the real-time gang is being scheduled on $k$ cores, all the $m-k$ cores are throttled according to the bandwidth threshold of the gang.

III-E Illustrative Example

In this subsection, we provide an illustrative example to compare scheduling under RT-Gang with a traditional co-scheduling approach.

Let us consider a multicore system with four homogeneous CPU cores. We want to schedule a parallel taskset, shown in Table I, in the system. $\tau_{1}^{RT}$ and $\tau_{2}^{RT}$ are real-time tasks. $\tau_{3}^{BE}$ is a best-effort task, which is scheduled under the default CFS scheduler. For the sake of simplicity, we assume that all threads of a task have the same compute time and are released simultaneously at the start of the period. We assume that all threads are statically pinned to specific CPU cores and they do not migrate during their execution. The OS scheduler tick interval is assumed to be 1-msec.

Figure 4(a) shows the scheduling timeline under a traditional co-scheduling approach. For this figure, we assume that tasks on different cores do not suffer interference due to contention in shared hardware resources. Under co-scheduling, $\tau_{1}^{RT}$ completes its execution at 2-msec. This leaves 8-msec slack duration on the two cores on which this task was executing. Similarly, $\tau_{2}^{RT}$ leaves a slack duration of 6-msec on its cores. Considering the system as a whole, the total slack duration left in this schedule is 28-msec. The slack duration can be used to schedule the best-effort task $\tau_{3}^{BE}$.

Figure 4(b) shows the scheduling timeline under RT-Gang. Under this schedule, $\tau_{1}^{RT}$ gets to run first. While $\tau_{1}^{RT}$ is executing, $\tau_{2}^{RT}$ is blocked thanks to our one-gang-at-a-time policy. Once $\tau_{1}^{RT}$ finishes its execution at 2-msec mark, $\tau_{2}^{RT}$ starts executing and completes at 6-msec mark. Under this scheme, the total slack duration left in the system is again 28-msec, assuming that $\tau_{3}^{BE}$ is not throttled (i.e., its memory bandwidth usage is less than the allowed threshold).

Now we consider the case where the real-time tasks can destructively interfere with each other due to resource contention. Borrowing from the DNN case-study introduced in Section II, we assume that the execution time of $\tau_{1}^{RT}$ increases 10X, when it is co-scheduled with $\tau_{2}^{RT}$. $\tau_{2}^{RT}$, on the other hand, stays unaffected under co-scheduling. Figure 4(c) shows the scheduling timeline for this case under co-scheduling scheme. As can be seen from the figure, while $\tau_{2}^{RT}$ is executing, the progress of $\tau_{1}^{RT}$ would be slowed due to interference. At 4-msec mark when $\tau_{2}^{RT}$ finishes its execution, $\tau_{1}^{RT}$ has only made 20% progress and it still has 1.6-msec of its original compute time left. For this reason, $\tau_{1}^{RT}$ completes its execution at 5.6-msec. In this case, the total slack duration for best-effort tasks is 20.8-msec.

Under RT-Gang, the scheduling timeline of the real-time tasks remains the same as the one shown in Figure 4(b) because $\tau_{1}^{RT}$ and $\tau_{2}^{RT}$ can never run at the same time. In other words, regardless of task and hardware characteristics, real-time tasks’ execution times would remain the same. The slack duration remains unchanged as well, which can be utilized for scheduling best-effort tasks although they are strictly regulated with a memory bandwidth throttling mechanism, shown as the “locked” duration in Figure 4(b).

In summary, the major benefits of RT-Gang are: (1) Parallel real-time tasks enjoy highly predictable task WCETs regardless of the characteristics of scheduled tasks and the underlying hardware because only one real-time task is scheduled at a time; (2) Best-effort tasks can be safely scheduled on any idle cores because the integrated throttling mechanism guarantees bounded interference.

IV Implementation

In this section, we describe the implementation details of RT-Gang in Linux kernel.

Algorithm 1 shows the high-level pseudo-code of RT-Gang. The implementation revolves around a data-structure struct glock declared in Line-2 of the algorithm. This data-structure is used for the following main purposes:

• •

Quickly check whether the gang scheduling lock is currently being held (held_flag)

• •

Track the cores, which are currently running real-time threads using a bitmask (locked_cores)

• •

Track the blocked cores, which have real-time tasks in their queues but cannot execute them due to the gang scheduling policy (blocked_cores)

To simplify the implementation complexity, we assume that each real-time gang in our system has a distinct real-time priority value. In the following sections, we explain the main parts of the algorithm in detail.

IV-A Gang Lock Acquisition

Before a real-time task can get scheduled, it needs to acquire the gang scheduling lock. Algorithm 2 shows the pseudo-code of the lock acquisition function. In this function, the gang-scheduling lock is marked as held by setting the flag in the global glock data-structure. The CPU core on which this function is invoked, is marked “locked” by setting its bit inside the locked_cores bitmask. The task that acquires the gang lock is marked as the gang-leader and its task pointer is tracked inside an array, which is later used at the time of lock release.

IV-B Gang Lock Release

This function is called to release the gang-scheduling lock on behalf of a task which is going out of schedule. The pseudo-code of this function is shown in Algorithm 3. Upon entering this function, it is checked whether the thread going out of execution is part of the currently executing gang. If this condition is true, the current CPU core is marked as unlocked, by clearing its bit from the locked_cores bitmask.

The next condition checked in this function is to make sure if all of the threads belonging to current gang have finished their execution, which would imply that the gang-lock is now free. This is done by checking if the locked_cores bitmask is zero. If this is the case, the gang-scheduling lock is marked free and rescheduling inter-processor interrupt (IPI) is sent to the CPU cores, which have blocked real-time tasks in their ready queues by using the blocked_cores bitmask.

IV-C Gang Preemption

The purpose of this function is to preempt all threads, which are part of the currently running gang, so that a new higher priority gang may start its execution. The pseudo-code of this function is shown in Algorithm 4. In this function, the locked_cores bitmask is traversed to send rescheduling IPIs to all the cores, which are currently marked as locked. Once this is done, the locked_cores bitmask is cleared and the threads being tracked in the gthreads array are removed.

IV-D Main Gang Scheduling Algorithm

The gang-scheduling algorithm resides in the critical path of the main scheduler entry point function (__schedule) in Linux and it works by modifying the task selection heuristics of the real-time scheduling class.

Algorithm 1 shows the main scheduling function of RT-Gang. The algorithm starts by checking whether gang-scheduling lock is currently being held by any real-time task. If that is the case, the algorithm tries to release the gang-scheduling lock on behalf of the prev task which is going out of schedule (Line-11).

If the gang-scheduling lock is currently free, the algorithm acquires the lock on the current core on behalf of the next real-time task (Line-13).

If the lock is currently not free, the algorithm checks whether the next real-time task on this CPU core belongs to the same gang that is holding the gang-scheduling lock (Line-14). Here, we exploit the fact that each real-time gang has a distinct priority as mentioned earlier this section. Hence, if the next task and the gang leader task have the same real-time priority value, they are considered a part of the same gang. In this case, the current core is marked “locked” by setting its bit in the locked_cores bitmask (Line-15).

If the lock is not free and the next real-time task does not belong to the currently executing gang, it is checked whether this task has a higher priority than the gang in execution (Line-16). If that is the case, the currently executing gang is preempted and the gang-scheduling lock is acquired on behalf of the next task (Line-17).

If all of the above conditions fail—i.e., the gang-scheduling lock is not free, the next real-time task does not belong to the currently executing gang, and it is of lower priority, then the next task is deemed ineligible for scheduling. In this case, the current CPU core is marked as blocked by setting its bit in the blocked_cores bitmask (Line-19) and the next task pointer is set to null so that no real-time task is returned to the scheduler by the real-time scheduling class.

Finally, the spinlock is released (Line-20) and control is returned to the scheduler (Line-21); to either schedule the next real-time task (if any) or go to the next scheduling class (CFS) to pick a best-effort task.

IV-E Creation of Virtual Gangs

In our implementation of RT-Gang in Linux, creation of virtual gang tasks is straight-forward. Because each real-time gang task has a distinct real-time priority in our system, the only thing that a system programmer has to do to mark different real-time tasks as part of the same virtual gang, is to assign them the same real-time priority value under Linux. Once this is done, RT-Gang framework allows simultaneous execution of such tasks just like real gangs.

IV-F Memory Bandwidth Throttling of Best-Effort Tasks

We incorporated a memory bandwidth throttling framework based on [55] in RT-Gang. This framework provides system programmers with a system-call; to mark the entire duration of a real-time application as memory bandwidth sensitive (i.e., Coarse-Lock). We update this system call such that instead of providing a binary value to start/stop throttling, the caller provides the acceptable memory threshold value for the real-time gang in execution. This value is stored in the task structure of the RT-Gang leader as an integer. We also modify the framework in [55] such that in every regulated interval, the memory bandwidth threshold value of the executing gang is automatically enforced on all CPU cores executing best-effort tasks. In this manner, we ensure that the real-time gang is protected from unbounded interference from best-effort tasks.

V Evaluation

In this section, we present the evaluation results of RT-Gang.

V-A Setup

We evaluate RT-Gang on two embedded platforms: Raspberry Pi3 and NVIDIA Jetson TX2. Raspberry Pi3 is equipped with a Cortex-A53 based quad-core CPU, which is representative of an energy efficient low-cost embedded multicore processor, while NVIDIA Jetson TX2 is equipped with a six-core heterogeneous CPU (4X Cortex-A57 and 2X Denver 111We do not use the Denver complex in our experiments due to its lack of hardware counter support needed to implement throttling mechanism [3]), which is representative of a high-end embedded processor.

On both platforms, we use Linux 4.4 kernel and implement RT-Gang by modifying its real-time scheduler (kernel/sched/rt.c). Our modification is about 500 lines of architecture neutral C code.

In all our experiments, we place the platform in the maximum performance mode and disable the dynamic frequency scaling of CPU cores. We also shutdown the graphical user interface and disable networking to minimize run to run variation in the experiments.

V-B Synthetic Workload

In this experiment, we show the benefits of RT-Gang using a synthetic taskset on Raspberry Pi3. The taskset is composed of two periodic multi-threaded real-time tasks ($\tau_{1}$ and $\tau_{2}$) and two single-threaded best-effort tasks ($\tau_{mem}^{BE}$ and $\tau_{cpu}^{BE}$). Using this taskset, we explore task execution time impacts to the real-time tasks and throughput impacts to the best-effort tasks.

We use a modified version of the BwRead benchmark—which sequentially accesses a 1-D array of a chosen size—from the IsolBench [51] benchmark suite to construct the taskset 222Our additions include support for multi-threaded and periodic task invocation. The code can be found in the IsolBench repository [1].. Concretely, $\tau_{1}$ creates two threads and is configured to use three quarters of the last-level cache size (384KB out of 512KB L2 cache of the Pi 3) as its working-set (which is shared between the two threads). It is periodically released at a 20ms interval and each job takes about 3.5ms in isolation. Similarly, $\tau_{2}$ is also a dual-threaded periodic real-time task with the same working-set size (384KB), but differs in its period (30 ms) and job execution times (6.5ms in isolation). We set the priority of $\tau_{1}$ higher than that of $\tau_{2}$, and schedule $\tau_{1}$ on Core 0,1 and $\tau_{2}$ on Core 2,3 (pinned using CPUSET interface).

Note that $\tau_{1}$ and $\tau_{2}$ are scheduled by SCHED_FIFO real-time scheduler with and without RT-Gang enabled 333RT-Gang can be enabled or disabled at runtime via: /sys/kernel/debug/sched_features. On the other hand, $\tau_{mem}^{BE}$ and $\tau_{cpu}^{BE}$ are both single-threaded best-effort tasks, which are scheduled by the CFS scheduler [35]; they differ in that $\tau_{mem}^{BE}$’s working-set size is two times bigger than the L2 cache size, while $\tau_{cpu}^{BE}$’s working-set is smaller than the core’s private L1 cache size. Thus, $\tau_{mem}^{BE}$ may cause interference to co-scheduled real-time tasks (if any) on different cores, due to contention in the shared L2 cache, while $\tau_{cpu}^{BE}$ may not. We collect the execution traces of the taskset without and with RT-Gang for a duration of 10 seconds using the trace-cmd and KernelShark [40].

Figure 5 shows the execution traces. In inset (a), during the first 20 msec duration, $\tau_{1}$ and $\tau_{2}$ were overlapped with each other, which resulted in significant job execution time increase for both tasks because their working-sets could not fit into the shared L2 cache simultaneously. During the next 20 msec duration, although $\tau_{1}$ and $\tau_{2}$ did not overlap, $\tau_{1}$ was overlapped with the two best-effort tasks, $\tau_{mem}^{BE}$ and $\tau_{cpu}^{BE}$, which also resulted in a significant increase in $\tau_{1}$’s job execution time (albeit to a lesser degree).

In inset (b), on the other hand, RT-Gang almost completely eliminates job execution time variance. In the first 20 msec duration, $\tau_{1}$ is overlapped with the two best-effort tasks. However, $\tau_{mem}^{BE}$ (the memory intensive one) was throttled most of the time, which protected the execution time of $\tau_{1}$. At around 40ms in the timeline, $\tau_{1}$ preempted $\tau_{2}$, the real-time task. In place of $\tau_{2}$, two best-effort tasks were scheduled, of which $\tau_{mem}^{BE}$ was again throttled as per $\tau_{1}$’s specified memory bandwidth threshold setting.

Note that in RT-Gang, the execution times of $\tau_{1}$ is deterministic. Furthermore, $\tau_{2}$ is also highly predictable as we only need to consider the preemption by the higher priority $\tau_{1}$, according to the classic response time analysis (RTA)[4]. Furthermore, because the two real-time tasks do not experience significant execution time increases, there are more “slacks” left for the best-effort tasks—compared to without using RT-Gang in inset (a)—which improves throughput of the best-effort tasks.

V-C DNN Workload

To establish the practicality of RT-Gang for real-world safety critical applications, we used the DNN workload introduced in Section II and executed it under different configurations on Raspberry Pi3 and NVIDIA Jetson TX2, with and without RT-Gang. The Cortex-A53 cores in Raspberry Pi-3 are much less capable than the four Cortex-A57 cores in Jetson TX2 platform. Moreover, the memory system in Raspberry Pi3 offers significantly less bandwidth than the one in Jetson TX2. Consequently, co-scheduled workloads in Raspberry Pi-3 are much more prone to the negative effects of resource contention. By evaluating RT-Gang on these two disparate platforms, we demonstrate its applicability to the range of embedded devices available today.

The taskset parameters for this experiment are shown in Table II. First, we use a multi-threaded DNN application as the high priority periodic real-time task ($\tau_{dnn(c)}^{RT}$ where $c$ denotes the number of cores used). The period of DNN inference operation is selected to keep the per-core utilization of DNN threads close to 45%. Second, as a lower priority real-time task, we use a periodic multi-threaded BwWrite benchmark ($\tau_{bww}^{RT}$). The working set size of BwWrite was chosen to be twice the size of LLC in each platform so as to stress memory subsystem (due to cache misses). The compute time ($C$) of BwWrite is selected to keep its per-core utilization less than 50% in all experiments. Lastly, we use two benchmarks from Parboil suite [48], lbm ($\tau_{lbm}^{BE}$) and cutcp, ($\tau_{cutcp}^{BE}$)) as best-effort tasks, which represent memory and CPU intensive parallel OpenMP applications respectively.

We vary the thread count (= # of assigned CPU cores) of the DNN task while keeping the thread count of $\tau_{bww}^{RT}$ and the best-effort tasks ($\tau_{lbm}^{BE}$ and $\tau_{cutcp}^{BE}$) fixed at four. For the experiment, we first measure the performance of $\tau_{dnn(c)}^{RT}$ in isolation, then co-scheduled with the rest of the taskset on baseline Linux, and finally using RT-Gang.

Figure 6 shows the cumulative distribution function (CDF) of the per-frame DNN inference time in each configuration (Solo: alone in isolation, Co-Sched: co-scheduled under baseline Linux, RT-Gang: co-scheduled under RT-Gang enabled Linux). Note first that execution times of the DNN task vary significantly under the co-scheduling scheme (Co-Sched). On Raspberry Pi3, the WCET across all configurations is more than 2X of its solo WCET. On TX2, the co-scheduling graphs again show deteriorated performance, albeit to a lesser extent than Raspberry Pi3.

Under RT-Gang, on the other hand, the execution times of the DNN workload are highly deterministic and match closely with its solo execution times in all tested configurations on both platforms. On Raspberry Pi3, however, although DNN’s performance is deterministic (i.e., low variance), noticeable performance constant increase is observed under RT-Gang when compared to the solo execution of the DNN task. We believe that this is caused by Cache Related Preemption Delay (CRPD) [27], as the memory system of Raspberry Pi3 is significantly less powerful than that of Jetson TX2.

Note that taking CRPD into analysis is well known in the context of single-core processors, but its applications have been difficult in multicore due to cache contention among the co-scheduled tasks on different cores, as shown in the CDF plots of Co-Sched. The determinism that RT-Gang brings thus would make CRPD analysis valid on multicore processors again, enabling effective timing analysis.

V-D Overhead

There are two main sources of overhead in our implementation of RT-Gang: 1) The serialization overhead associated with the critical section of our gang scheduling algorithm in selecting next real-time task. 2) The overhead involved in sending cross-core interrupts (IPIs) and acquiring ready-queue locks for gang preemption.

The serialization overhead of RT-Gang is only incurred during the selection of real-time tasks due to the use of a spinlock. However, the length of the critical section of RT-Gang is small—comparable to existing spinlocks used in the various parts of the Linux kernel scheduler. On the other hand, the overhead associated with gang-preemption due to the IPIs can pose a scalability problem as the number of necessary IPIs can be as many as all the rest of the cores.

In order to estimate both these overheads, we conducted an experiment on the NVIDIA Jetson TX2 platform in which a high priority real-time gang preempts a multi-threaded low priority real-time gang, a fixed number of times ($100000$), with and without RT-Gang. We also varied the number of threads of the low priority gang to see the effect of gang size on the gang preemption overhead. The result from this experiment is shown in Table III.

As can been seen from the table, RT-Gang adds very small overhead to the overall cost of a context-switch under Linux; considering the fact that for a well-designed system, a context-switch is not supposed to happen too frequently. The findings from this experiment also match the results seen during evaluation with DNN workloads; in which, we saw that the performance of these workloads remain completely unaffected under RT-Gang.

VI Discussion

In this section, we briefly discuss potential use-cases of RT-Gang. We believe that our simple design and practical implementation leveraging existing real-time scheduler in Linux offer broader practical use-cases in many application domains that concern timing predictability and need to run parallelized multi-thread applications. Time critical parallel real-time applications in automotive and aviation domains (e.g., perception and control applications in a self-driving car) are our main target use-cases. Also, barrier based scientific applications in high-performance computing (HPC) can potentially benefit from using RT-Gang as they are sensitive to thread imbalance, and thus motivated original gang scheduling research [18] in the first place. Although we mainly target embedded multicore processors (typically having 4-8 cores) in this work, we recently were able to apply our RT-Gang kernel patch on a 12 hardware thread (6 core) x86 PC, successfully performing gang scheduling across the 12 hardware threads. Effectiveness and scalability of RT-Gang in other application domains, such as HPC, is left as future work.

VII Related Work

Parallel Real-Time Scheduling. Parallel real-time tasks are generally modeled using one of following three models: Fork-join model [26, 42, 36, 13], dag model [6, 41] and gang task model [19, 23, 17]. In the fork-join model, a task alternates parallel (fork) and sequential (join) phases over time. In the dag model, a task is represented as a directed acyclic graph with a set of associated precedence constraints, which allows more flexible scheduling as long as the constraints are satisfied. Lastly, in the gang model, a task is simply represented with an execution time $e$ and a number of cores $k$ it needs to run. While most simple among the three, it matches well with real-world parallel applications where users or scheduler selects the number of threads an application might use at the start time, which is also the case in DNN workload we used in Section II.

Our one-gang-at-a-time scheduling policy is essentially a restrictive form of gang scheduling [18]. In traditional gang scheduling, all threads of an application are scheduled simultaneously, and more than one application is allowed to be scheduled as long as there are available cores. In contrast, our approach is restrictive in the sense that such co-scheduling is not allowed. Gang scheduling was originally studied in high-performance computing as a way to maximize performance of parallel applications [18] or virtual machine scheduling [14]. More recently, gang scheduling was investigated as a way to improve security [46], by preventing simultaneously co-scheduling different security domains, similar to our one-gang-at-a-time policy, but it was implemented in Linux’s CFS scheduler and thus does not support real-time task scheduling.

In the real-time systems community, fixed-priority and dynamic priority real-time versions of gang scheduling policies, namely Gang FTP and Gang EDF, respectively, are studied and analyzed [19, 23, 17]. However, these prior real-time gang scheduling policies also allow co-scheduling of multiple gangs as long as there exist available cores because they mainly focus on CPU utilization, without considering the negative WCET impact of co-scheduling on shared memory multicore platforms. On the other hand, the Isolation Scheduling model [21] and the integrated modular avionic (IMA) scheduler design in [34] consider shared resource interference and limit co-scheduling to the tasks of the same criticality (in  [21]) or those in the same IMA partition (in [34]). However, they do not specifically target parallel real-time tasks and do not allow co-scheduling of best-effort tasks. Also, to our knowledge, all aforementioned real-time scheduling policies were not implemented in actual OSes. Our work is, to the best of our knowledge, the first real-time gang scheduler implementation in Linux, which implements a fixed-priority real-time gang scheduling, enforcing the one-gang-at-a-time policy. Furthermore, we enable safely co-scheduling best-effort tasks, by integrating an OS-level throttling mechanism [55], to improve system utilization when there are best-effort tasks—a common situation in practice.

Real-Time Scheduling in Linux. A number of efforts in the real-time systems community have been aimed at improving real-time scheduling in Linux. For example, $LITMUS^{RT}$ [10] developed a set of global, partitioned, and semi-partitioned real-time schedulers, outside of the existing Linux’s real-time scheduler; ChronOS Linux developed global and local scheduling layers built on top of Linux’s O(1) scheduler [15]. In contrast, RT-Gang is developed as an extension (a feature) of Linux’s default real-time scheduler that enforces a simple invariant—one real-time gang across all cores—otherwise respecting standard fixed-priority real-time scheduling, and does not maintain its own scheduling queues, unlike the aforementioned two prior implementations. We believe our design is simpler and easier to maintain in the face of rapid change in Linux kernel.

OS-level Shared Resource Partitioning. Many researchers have attempted to make COTS multicore platforms to be more predictable with OS-level techniques. A majority of prior works focused on partitioning of shared resources among the tasks and cores to improve predictability. Page coloring has long been studied to partition shared cache [30, 31, 58, 45, 16, 53, 24, 54], DRAM banks [56, 32, 50], and TLB [38]. Some COTS processors [22, 33] support cache-way partitioning [49]. Mancuso et al. [33] and Kim et al. [25], used both coloring and cache way partitioning for fine-grained cache partitioning. While these shared resource partitioning techniques can reduce space conflicts of some shared resources, hence beneficial for predictability, but they are not enough to guarantee strong time predictability on COTS multicore platforms because there are too many hardware resources (e.g., cache MSHRs, DRAM controller buffers, etc.) that have profound impact on task timing [51, 8], but are unpartitionable and out of control of software.

VIII Conclusion

We presented RT-Gang: a novel real-time gang scheduling framework for predictable and efficient parallel real-time scheduling on multicore.

RT-Gang implements a novel gang scheduling policy that eliminates inter-task interference by enforcing an invariant that only one (parallel) real-time task (gang) can be scheduled at any given time. This enables tighter task WCET estimation and simpler schedulability analysis. RT-Gang also provides additional mechanisms, namely virtual gang and best-effort task throttling, which can help maximize system utilization while providing strong time predictability to real-time tasks.

We implemented RT-Gang in Linux and evaluated it on two embedded multicore platforms. The evaluation results show the predictability and efficiency benefits of RT-Gang. In future, we plan to extend our RT-Gang approach to heterogeneous multicore platforms.

Acknowledgements

This research is supported by NSF CNS 1718880, CNS 1815959, and NSA Science of Security initiative contract #H98230-18-D-0009.