Zygarde: Time-Sensitive On-Device Deep Inference and Adaptation on Intermittently-Powered Systems

TL;DR

Zygarde is a novel scheduling framework for batteryless, intermittently-powered systems that balances energy, accuracy, and real-time constraints to improve deep learning task execution on microcontrollers.

Contribution

It introduces an energy- and accuracy-aware scheduling algorithm specifically designed for intermittently-powered systems executing deep neural networks.

Findings

Reduces DNN task execution time by up to 26%.

Schedules 9%-34% more tasks than traditional methods.

Achieves up to 21% higher inference accuracy.

Abstract

We propose Zygarde -- which is an energy -- and accuracy-aware soft real-time task scheduling framework for batteryless systems that flexibly execute deep learning tasks1 that are suitable for running on microcontrollers. The sporadic nature of harvested energy, resource constraints of the embedded platform, and the computational demand of deep neural networks (DNNs) pose a unique and challenging real-time scheduling problem for which no solutions have been proposed in the literature. We empirically study the problem and model the energy harvesting pattern as well as the trade-off between the accuracy and execution of a DNN. We develop an imprecise computing-based scheduling algorithm that improves the timeliness of DNN tasks on intermittently powered systems. We evaluate Zygarde using four standard datasets as well as by deploying it in six real-life applications involving audio and camera sensor systems. Results show that Zygarde decreases the execution time by up to 26% and schedules 9%-34% more tasks with up to 21% higher inference accuracy, compared to traditional schedulers such as the earliest deadline first (EDF).

Figures

Tables

Table 1. Table 1. Real-life evaluation setup.

Application	Energy Source	Harvester Placement	Cause of Intermittence	Target Event	Other Events
Car Detector	Solar (74Klx to 111Klx)	Pavement	Vehicle on the closest lane	Car Honk	Silence, Dog, Human Voice, Car
Barking Dog	Solar (2Klx to 18Klx)	Under the Tree	People, objects and cloud	Dog Bark	Silence, Car, Car Honk, Voice
People Detector	Solar (1Klx to 5 Klx)	Edge of the Railing	People and cloud	Voice	Silence, Car, Honk, Dog Bark
Baby Monitor	RF (-0.48dB to -1.66dB)	On the Desk	Change of Distance	Crying Baby	Silence, Voice, Washer, Printer
Laundry Monitor	RF (-0.48dB to -1.91dB)	On the Counter	Change of Distance	Washer Status	Silence, Voice, Cry, Printer
Printer Monitor	RF (-1.59dB to -1.91dB)	On the Desk	Change of Distance	Printer Status	Silence, Voice, Crying Baby, Printer

Table 2. Table 2. Classification Accuracy for Different Models.

Classifier	MNIST	ESC-10	CIFAR-100	VWW
KNN	92% (Ebrahimzadeh and Jampour, 2014)	40%	55%	60%
K-means	93% (Ebrahimzadeh and Jampour, 2014)	41%	50%	59%
Random Forest	93% (Ebrahimzadeh and Jampour, 2014)	25%	29%	62%
SVM	96% (Ebrahimzadeh and Jampour, 2014)	50%	51%	69%
CNN (No Early Termination)	98%	75%	78%	84%
CNN (Early Termination)	97%	73%	77%	84%

Equations

$$h(N)=\{\begin{array}{ll}p(H_t=1\mid H_{t-1}\wedge \dots \wedge H_{t-N}=1),& \text{for }N>0\\ p(H_t=1\mid H_{t-1}\vee \dots \vee H_{t-N}=0),& \text{for }N<0\end{array}$$

$$KW\big{(}\mathscr{H}^{(i)},\mathscr{P}\big{)}=\int_{-\infty}^{+\infty}|CDF(\mathscr{H}^{(i)})-CDF(\mathscr{P})|$$

$$\eta=1-\frac{KW\big{(}\mathscr{H}^{(i)},\mathscr{P}\big{)}}{KW\big{(}\mathscr{R},\mathscr{P}\big{)}}$$

$$\displaystyle\begin{split}LA=\sum_{i=1}^{L}a_{i}\times LC\Big{(}W^{i},X_{1}^{i},X_{2}^{i},\cdots,X_{N}^{i}\Big{)}\end{split}$$

$$\displaystyle\begin{split}LC\Big{(}W^{i},X_{1}^{i},X_{2}^{i}\Big{)}=\dfrac{1}{2}\Big{(}1-Y\Big{)}\Big{(}G_{W^{i}}(X_{1}^{i})-G_{W^{i}}(X_{2}^{i})\Big{)}+\dfrac{1}{2}Y\max\Big{(}0,\Delta-G_{W^{i}}(X_{1}^{i})-G_{W^{l}}(X_{2}^{i})\Big{)}\end{split}$$

$$\zeta_{i,j}^{l}=\Big{(}1-\alpha(d_{i,j}-t_{c})\Big{)}+\Big{(}1-\beta\Psi_{i,j}^{l}\Big{)}+\gamma_{i,j}^{l}$$

$$\zeta_{I_{i,j}}^{l}=\begin{cases}\big{(}1-\alpha(d_{i,j}-t_{c})\big{)}+\big{(}1-\beta\Psi_{i,j}^{l}\big{)}+\gamma_{i,j}^{l},&\eta E_{curr}\geq E_{opt}\\ \gamma_{i,j}^{l}\Big{(}\big{(}1-\alpha(d_{i,j}-t_{c})\big{)}+(1-\beta\Psi_{i,j}^{l})\Big{)},&\eta E_{curr}<E_{opt}\end{cases}$$

Full text

Intermittent Computing, Deep Neural Network, Semi-Supervised Learning, On-Device Computation, Real-Time Systems, Batteryless System, Energy harvesting System, On-Device Learning

††copyright: acmcopyright††journal: IMWUT††journalyear: 2020††journalvolume: 4††journalnumber: 3††article: 82††publicationmonth: 9††price: 15.00††doi: 10.1145/3411808††ccs: Computing methodologies Machine learning††ccs: Computer systems organization Embedded systems††ccs: Hardware Power and energy††ccs: Computer systems organization Real-time system specification

1. Introduction

Batteryless IoT devices are powered by harvesting energy from ambient sources, such as solar, thermal, kinetic, and radio-frequency signals (RF). These devices, in principle, last forever—as long as the energy harvesting conditions are in favor. They have applications in long-term sensing and inference scenarios, such as wildlife monitoring, remote surveillance, environment and infrastructure monitoring, and health monitoring using wearables and implantables. While every IoT application has an expected response time, many of them require timely feedback. For instance, in acoustic sensing systems, such as hearables and voice assistants (Hoy, 2018; Islam et al., 2017), car detectors (de Godoy et al., 2018), and machine monitors (Srinivasan et al., 2017), events need to be detected and reported on time in order to ensure prompt responses, safety, and timely maintenance. Though a batteryless system is desirable, the unpredictability of the harvested energy, combined with the complexity of on-device event recognition tasks, complicates timely execution of machine learning-based event detection tasks on such systems.

Prior works on time-aware batteryless computing systems can be broadly categorized into two types. The first category focuses on maintaining a reliable system clock when the power is out (de Winkel et al., 2020; Rahmati et al., 2012; Hester et al., 2016). The second category includes runtime systems that consider the temporal aspect of data across power failures (Hester et al., 2017; Buettner et al., 2011; Zhu et al., 2012; Yıldırım et al., 2018) by discarding data after a predefined interval or considering energy to increase the chances of task completion. Recent works (Gobieski et al., 2018, 2019; Lee and Nirjon, 2019) on batteryless systems have proposed frameworks and runtime to execute non-real-time DNN1 tasks on intermittently-powered systems. The real-time community have proposed techniques (Bateni and Liu, 2018; Zhou et al., 2018; Yang et al., 2019; Lee and Shahriar, 2020; Islam and Shahriar, 2020) for deadline-aware execution of DNNs, primarily on GPU and server-grade machines. Despite these commendable efforts, there is a gap in the existing literature that none has considered all three dimensions, i.e., intermittence of harvested energy, variable utility of DNN inference, and real-time schedulability; and merely combining existing solutions does not entirely solve the problem.

We illustrate this using an example in Figure 1. We consider two jobs, $J_{1}$ and $J_{2}$, released at time 0 and 20, respectively. Their relative deadline is 34, and the execution time is 28. The intermittency of energy generally has no consistent pattern in the duration of or in the gaps between ON/OFF phases. Figure 1(a) shows that when the power is uninterrupted, both jobs meet their deadlines under the earliest deadline first (EDF) scheduling. When power is intermittent (Figure 1(b)), task $J_{2}$ misses its deadline. Figure 1(c) illustrates our proposed approach which partitions a job into mandatory and optional portions and ensures that the mandatory portion (which is required to achieve a desirable inference accuracy) of each job finishes on time. And if there is extra time, our proposed approach schedules some optional jobs that may increase the accuracy further.

In this paper, we present Zygarde— which is the first system that enables deadline-aware imprecise execution of DNNs on an intermittently-powered system. The design of Zygarde is motivated by two observations. First, energy generated by a harvester is bursty, i.e., the state of energy generation is maintained over a short period. This property enables us to obtain a probabilistic model of the energy harvesting pattern – which can be characterized by a single parameter for a given harvester used in a particular application scenario. Second, since deeper layers of a DNN extract more detailed and fine-grained features of the input data, for a given accuracy, the required amount of DNN computation depends on the quality of data itself. Zygarde exploits these observations and proposes an imprecise computing-based (Shih and Liu, 1992; Liu et al., 1991) online scheduling algorithm, which considers both the intermittence of energy and the accuracy-execution trade-off of a DNN.

The main contribution of this paper is a deadline-aware runtime framework for DNNs executing on intermittently-powered systems, for which, runtime adaptation of a DNN is necessary, on top of compile-time compression (Gobieski et al., 2019; Han et al., 2015; Nabhan and Zomaya, 1994; Yao et al., 2017; Yao et al., 2018a, b, [n.d.]). Compile-time compression alone is not sufficient when the remaining deadline is inadequate for a full execution of the DNN, but long enough to compute the inference result from the partial execution of the DNN. This runtime adaptation process requires (1) modeling the energy harvesting pattern using a single factor($\eta$), which helps determine the system how much computation is possible in the near future, and (2) specialized offline training of the DNNs to minimize the loss of accuracy due to partial execution.

Zygarde complements prior work on batteryless systems such as time-keeping (Rahmati et al., 2012; Hester et al., 2016) and execution of non-real-time tasks (Gobieski et al., 2018, 2019; Ransford et al., 2012; Mirhoseini et al., 2013; Balsamo et al., 2015, 2016; Maeng et al., 2017; Colin and Lucia, 2016; Maeng and Lucia, 2018). In contrast to all previous works, Zygarde’s contribution is at the framework, modeling, and algorithmic level, while its implementation relies upon existing open-source frameworks and APIs (Gobieski et al., 2019; Maeng et al., 2017) that handle the lower-level aspects of an intermittent system.

We implement Zygarde on a TI MSP430FR5994 microcontroller and evaluate its performance using four standard datasets (MNIST (LeCun et al., 1998), ESC-10 (Piczak, 2015), CIFAR-100 (Krizhevsky et al., 2009), and Visual Wake Works (Chowdhery et al., 2019)) as well as in six real-world acoustic event detection and visual sensing experiments. Zygarde achieves 5%–26% reduction in execution time using early termination. By using the layer-aware loss for early termination, it also increases the inference accuracy by upto 21% than the state-of-the-art solutions that use cross-entropy loss  (Zhang and Sabuncu, 2018) and contrastive loss (Islam and Nirjon, 2019). Moreover, it schedules 9%–34% more jobs with upto 30% higher inference accuracy than the earliest deadline first (EDF) scheduling algorithm. Furthermore, it gains up to 28% higher inference accuracy than the imprecise variant of EDF.

2. Zygarde System Design

In this section, we describe the system architecture of Zygarde and provide an example to illustrate its execution.

2.1. System Architecture

Zygarde is a framework and a runtime system that enables time-aware execution of a DNN on intermittently-powered systems. It consists of five major components: a job generator, an energy manager, an agile DNN model, semi-supervised $k$-means classifiers, and a scheduler. These components are shown in Figure 2.

Job Generator. Zygarde reads data from sensors (e.g., a microphone) and writes them to the non-volatile memory (FRAM) of the microcontroller using direct memory access (DMA). It considers the processing of the sensor data stream for each classification task (e.g., user identification and event detection) as separate tasks. The end-to-end processing of one data sample of a sensor stream (i.e., from feature extraction to classification and model adaptation) is called a job. For example, an audio sensing system that performs both speaker recognition and hotword detection has two tasks, and the processing of each audio frame to recognize the speaker and to detect the hotword are two separate jobs. Thus, if this system (having 2 tasks) generates $k$ audio frames/second, then after 3 seconds, there will be a total of $6k$ jobs. The Job Generator creates and enqueues jobs into a queue. A job leaves the queue when it gets scheduled for execution or its deadline has passed.

Energy Manager. The Energy Manager monitors the state of the energy storage (i.e., the supercapacitor) and the energy harvesting rate. The scheduler uses this information for scheduling decisions (described in Section 5). The energy manager implements an open-source intermittent computing runtime (Gobieski et al., 2019; Maeng et al., 2017) to manage the execution of jobs across power failures. At the implementation level, each job of Zygarde consists of multiple small atomic fragments that maintain a strict precedence order. These fragments execute atomically and the runtime ensures that repeated attempts to execute a fragment is idempotent.

Agile DNN Model. The Agile DNN Model is a pre-trained deep neural network that converts data samples into feature representations (Bengio et al., 2013). The network is trained offline, on a high-end server, using labeled training data. We use rank-decomposition (Bhattacharya and Lane, 2016) and separation (Han et al., 2015) to compress and fit the network into the limited memory of a microcontroller. Based on the quality of the input data sample, Zygarde may terminate the execution of the DNN early at runtime, and hence, we call it an agile DNN Model. We note that an agile DNN is a special type of anytime DNN (Teerapittayanon et al., 2016; Yu and Huang, 2019; Karayev et al., 2012; Karayev et al., 2014; Hu et al., 2019; Kim et al., 2018) — where the depth of a neural network is dynamically adjusted during inference. However, unlike anytime networks, where the output of a hidden layer is fed to a secondary shallow neural network for classification, agile DNN employs a cluster-based classifier to replace expensive matrix multiplication operations with 4X less costly additions/subtractions (add, 2020; iqM, 2015). In our implementation of agile DNN, this design saves 27,750 execution cycles per inference, when compared to anytime neural networks.

Semi-Supervised $k$-Means Classifiers. The feature representation obtained from the execution of the Agile DNN is classified by a semi-supervised $k$-means clustering algorithm (Arthur and Vassilvitskii, 2006). The clustering algorithm uses the L1 distance between two feature vectors (Black, 1998). For layers (e.g., the convolution layers) that produce two or more dimensional features, they are flattened or vectorized prior to computing the L1 norm. Since the execution of an Agile DNN may terminate at any layer depending on the input data, Zygarde maintains a separate $k$-means classifier for each layer of the Agile DNN. These $k$-means classifiers are trained offline on a server machine. However, to enable online learning, these classifiers are updated at runtime using a model adaptation process described in Section 4.3. The motivation behind the $k$-mean based approach is to reduce the execution time and energy consumption by avoiding multiplications which is over 4$\times$ more expensive than additions and subtractions.

Scheduler. Zygarde implements an online, dynamic priority, real-time scheduler that considers not only the timing aspects but also the expected inference accuracy of a job and the energy harvesting status of the system. It dynamically partitions an executing job into mandatory and optional portions based on the early termination of an agile DNN and prioritizes the execution of its mandatory portion to ensure both timeliness and accuracy under the constraints of intermittently available energy. Note that, the beginning portion of all jobs are mandatory and whether the next unit is mandatory or optional is determined during the execution of the current unit. An illustration of the scheduler is described next.

2.2. Example Execution

Table 5 defines two jobs $J_{1,1}$ and $J_{1,2}$ from the same task, $\tau_{1}$. Figure 5 demonstrates the execution of the jobs along with the status of harvested energy over a timeline. $E_{curr}$ refers to the current energy, and $E_{opt}$ and $E_{man}$ refer to two thresholds that determines whether the optional and the mandatory parts of a job should be executed, respectively. $J_{i,j}^{k}$ refers to the $k^{th}$ partition of job $J_{i,j}$. Table 5 shows the action taken by the scheduler along with the reasoning at each time step. Here, the mandatory part of $J_{1,2}$ is longer than the mandatory part of $J_{1,1}$ because, for the second data sample, the classifier is not confident enough with the result after the execution of the first layer. Note that to simplify the illustration, we assume that each layer requires one time unit to execute and the partition (mandatory vs. optional) is known ahead of time. The proposed scheduling algorithm (described in Section 5) handles further complexities such as the different execution times of different layers, multiple time units per layer, dynamic partitioning, and power-failure during the execution of a layer.

Setting $E_{man}$ and $E_{opt}$. The $E_{man}$ is set to the minimum energy required to turn on the MCU and execute an atomic fragment. During the compile time, Zygarde programming tools (described in Section 6) estimates the maximum energy required by any atomic fragment by running EnergyTrace++ (Ene, 2014) and sets this threshold. The $E_{opt}$, on the other hand, is by default set to the energy required to fill up the capacitor. This is because, once the capacitor is full, the excess energy gets wasted if nothing is executing on the MCU. By executing optional tasks, we minimize this wastage, and increase the performance of the system. However, a developer can override these values using the APIs provided (Section  6.2). If $E_{opt}$ is small, e.g., comparable or equal to $E_{man}$, then all the optional portion of the tasks will execute, causing starvation of the mandatory tasks. On the other hand, if $E_{opt}$ is high, the optional portion of the jobs will never execute.

3. Modeling Intermittent Energy

This section derives a single metric, namely the $\eta$-factor, which characterizes an energy harvester used in a particular application. The metric is later used in the real-time scheduler in Zygarde.

3.1. Energy Event

The unavailability of harvestable energy and the required time to buffer sufficient energy before it can be used cause intermittence in batteryless systems. The pattern of intermittence is stochastic, hard to predict, and heavily dependent on the available harvestable energy. Hence, instead of modeling and predicting the harvested energy to characterize a harvester, we define a binary random variable, called the energy event, that denotes the state of the energy storage to have at least $\Delta K$ Joules of energy over a $\Delta T$ period, where $\Delta K$ and $\Delta T$ depend on the application as well as the underlying system. In our empirical model, we set $\Delta K$ to $E_{man}$.

Furthermore, instead of directly dealing with the harvested energy, it is often easier to observe the physical phenomenon behind energy generation. For instance, for a piezoelectric harvester installed inside a smart shoe, the number of footsteps that generate at least $\Delta K$ Joules over $\Delta T$ time can be equivalently used to define an energy event. Likewise, a certain light-intensity for a solar harvester, and a certain number of packet transmissions for an RF harvester over $\Delta T$ time can be used to define energy events for these systems.

To characterize an energy harvester using the definition of energy events, we conduct a two-month long study on solar and RF harvesters (using empirically collected data) and human footsteps (using the dataset (Meyer et al., 2016)) to analyze the pattern of energy events. Our study reveals that energy events occur in bursts, i.e., every harvester has a tendency to maintain its current binary state, and there is a probabilistic relation between consecutive energy events over a period.

For instance, when a person starts walking, the probability that they will continue to walk is high over the next few time units and the probability decreases with time. Conversely, when a person is not walking, the probability that they will continue not to walk is going to be high over the next few time units and will diminish with time. This observation enables us to impose conditional probability on future energy events, given the recent history of energy events—which is the key to characterize an energy harvester. We denote an energy event at time $t$ using the random variable $H_{t}\in\{0,1\}$.

3.2. Conditional Energy Event

We define conditional energy event, $h(N)$ as the probability that an energy event will occur, given the immediately preceding $N$ consecutive energy events have occurred (for $N>0$) or have not occurred (for $N<0$):

[TABLE]

To illustrate, h(10) = 90% implies that an energy event will occur with 90% probability if 10 immediately preceding consecutive energy events have occurred.

Figures 6(a)-(d) show the distribution of $h(N)$ for a persistently-powered and three energy harvested systems. To characterize an energy harvester, we measure the Kantorovich-Wasserstein (KW) distance (Kantorovich, 1942) between its distribution, $\mathscr{H}^{(i)}$ from an ideal (persistent power) source, $\mathscr{P}$ to obtain:

[TABLE]

In Figure 6, we observe that $h(N)$ drops when $|N|$ increases. For example, in Figure 6(b), $h(N)$ drops after $N=20$ since the person we studied never walked for more than 100 minutes. Similarly, in Figure 6(c), after about five hours of consecutive energy events (i.e., light intensity ¿ 2730 lux), the probability of energy event drops as the stationary solar harvester was placed beside a window that does not get enough light after five hours. We also notice that after about 19 hours of absence of any energy event (i.e., light intensity ¡ 2730 lux), the probability of the next energy event is high as the sun shows up again at the window.

3.3. The $\eta$ Factor

Despite being informative, the KW distance has a limitation that not all $h(N)$’s are estimated using the same number of instances. Hence, we normalize the KW score against a purely random harvesting pattern, $\mathscr{R}$ to obtain a revised metric, called the $\eta$-factor:

[TABLE]

The value of $\eta$ lies in $[0,1]$ and it measures how close a harvester’s harvesting pattern is to a constant energy source. For a persistently-powered system, $\eta=1$, and for an energy harvester that shows no apparent pattern has $\eta=0$. For any other energy harvesting system, the $\eta$-factor will lie in-between and it is generally high for small $|N|$. A higher $\eta$-factor indicates less randomness in its energy harvesting pattern, and thus encourages a scheduler to make more aggressive decisions on scheduling tasks in the next few time slots. The $\eta$-factor needs to be empirically estimated for a given application-specific system. A critical discussion of $\eta$-factor is presented in Section 11.

4. Modeling DNN Tasks

In this section, we describe the task model of Zygarde, training procedure of agile DNN, and construction of semi-supervised $k$-means classifier.

4.1. Task Model

Tasks, Jobs, Units, and Fragments. Zygarde considers the processing of a sensor data stream for each classification task as an imprecise sporadic task (Liu et al., 1991), $\tau_{i}=(T_{i},D_{i},C_{i})$, where $T_{i}$ denotes the period (i.e., the minimum separation between two consecutive jobs), $D_{i}$ is the relative deadline, and $C_{i}$ is the worst-case execution time.

An instance of a task, aka a job, comprises of an ordered sequence of modules (henceforth these modules are called units). The first $M$ units are mandatory and must be completed before the deadline, whereas the rest of the units of a job are optional and can be executed if time and resources are available. Such a partitioning scheme is known as the imprecise computing model in real-time systems literature (Liu et al., 1991; Zhang et al., 2004; Moallemi and Wainer, 2011). In this paper, however, the partition (i.e., the value of $M$) is dynamic and depends on the input data.

In Zygarde, a job that executes an $L$-layer agile DNN has $L$ units, where each unit corresponds to processing one DNN layer, along with the execution of the corresponding semi-supervised $k$-means classifier. The cluster centroids of this semi-supervised $k$-means classifier are updated with new unlabeled data. Based on the input data, Zygarde may decide to exit from a unit or continue executing the next unit. The decision is based on a utility function, which is described next.

Although a unit represents a logical grouping of related modules at each layer of the DNN, the size of a typical unit is generally too large to execute without an intermittence. Hence, at the implementation level, to avoid corrupted results and to ensure forward progress of code execution, these units are further divided into atomically executable fragments—which guarantees correct intermittent execution using SONIC API (Gobieski et al., 2019).

Dynamic Partitioning and Utility. Unlike traditional imprecise computing models (Liu et al., 1991; Zhang et al., 2004; Moallemi and Wainer, 2011) where the partition of a job into mandatory and optional parts is known, the number of mandatory units in Zygarde is determined at runtime. We propose a utility function that estimates the confidence in classification at a given unit for that job. This represents the utility of the data where higher utility at earlier units is desirable.

Since we use a $k$-means classifier, we assume that a classification result is more likely to be correct if the input data sample being processed is unambiguously close to exactly one of the $k$ means. To achieve this, we compute the L1 distance of an input data sample (represented in terms a DNN layer and then vectorized) from two of its closest of the $k$ means, $\Delta_{1}$ and $\Delta_{2}$, and if their difference $|\Delta_{2}-\Delta_{1}|$ is above a unit-specific threshold, we decide to classify it as belonging to its closest cluster; otherwise, the computation of the DNN continues to the next unit. The process is illustrated by Figure 8.

The utility function described above runs in linear time with the number of clusters, i.e., $\mathcal{O}(k)$. It is lightweight, energy-efficient, and suitable for resource-constrained systems as it uses the byproduct of clustering-based classification which computes the cluster distances, $\Delta_{i}$’s anyways. It, however, depends on an offline-estimated threshold. Section 4.3 describes how the utility threshold is computed using an empirical dataset. Section 11 further discusses alternative utility functions that are suitable for other types of classifiers.

Preemption and Task Switching Zygarde allows limited preemption (Baruah, 2005) where a job can be preempted only after a unit completes its execution. The scheduler kicks in at the completion of a unit and at the deadline of a job. After the execution of a unit, a job returns to the job queue with updated utility and imprecise status (mandatory or optional). Then the scheduler chooses the next highest priority job from the job queue using the priority function described in Section 5. By prohibiting preemption of a unit, Zygarde reduces context switching and read-write overheads, and minimizes the memory requirements to $\mathcal{O}(N)$ for $N$ jobs by using double-buffering  (Asaro et al., 2000). Figure 8 shows the execution of two units. Each unit is shown as a large dotted rectangle and it contains four logical modules that are shown as solid rectangular boxes.

4.2. Agile DNN Construction

Unlike previous works where inference happens only at the last layer (Yang et al., 2016) or where a second classifier is used at hidden layers to decide early exit (Teerapittayanon et al., 2016), in Zygarde, the output of any hidden layer can be directly used as a feature that gets classified by a k-means classifier. Features obtained in this manner neither guarantee that the data samples from the same class are closer nor guarantee that the data samples from different classes are farther in the feature space. Hence, to ensure that the feature representation obtained after an early exit from the DNN execution maximizes the separability of different classes and minimizes the distance between examples of the same class, Zygarde employs a layer-aware loss function.

**Layer-Aware Loss Function. ** A convex combination of contrastive losses (Islam and Nirjon, 2019) at each layer is used as the loss function of an Agile DNN – which we call the layer-aware loss function:

[TABLE]

where, $a_{i}$ is the convex coefficient for the $i^{th}$ layer and $\sum_{i=1}^{L}a_{i}=1$; $L$ and $N$ represent the total number of layers and classes, respectively; $W^{i}$ represent the weights of the $i^{th}$ layer; $X_{1}^{i},X_{2}^{i},\cdots,X_{N}^{i}$ are the output vectors corresponding to the members of each class at the $i^{th}$ layer; and $LC$ is the contrastive loss function. For two classes, $LC$ is defined as:

[TABLE]

where, $G_{W^{i}}(X_{n}^{i})$ is the output feature set of a member of the $n$-th class ($1\leq n\leq N$) at layer $i$. The coefficient $Y=0$, if $X_{1}$ and $X_{2}$ belong to the same class, and $Y=1$, otherwise. $\Delta$ represents the margin between the members of different classes in the feature space.

Training Agile DNN. To train an agile DNN, we use a siamese network architecture (Islam and Nirjon, 2019) as shown in Figure 10. In a siamese network, there are two identical neural networks, called the sister networks, that share the same weights. From the labeled dataset, we select pairs of data points and use them as the inputs to the twin networks. Among the selected pairs of data points, 50% belong to the same class, while the rest belong to different classes. Unlike (Islam and Nirjon, 2019), which only uses the contrastive loss at the last layer ($LC_{3}$), we use the layer-aware loss function (Equation 4) at every layer to train these networks. We perform an exhaustive search for hyper-parameter tuning and to determine the weights of each layer. After the training, we use only one of the sister networks for inference. During inference, we obtain a representation of the input data from each layer, and use them as features for the semi-supervised $k$-means classifiers. Note that, for convolution layers, we flatten the output of a layer to get a vector instead of a tensor in order to be able to compute the L1-norm during the clustering-based classification step of Zygarde.

Note that although the combination of an agile DNN and semi-supervised $k$-means classifiers in Zygarde is inspired by Anytime Neural Networks, an agile DNN is different as it is a representation learner rather than a classifier. Besides, an agile DNN is trained using a siamese network and it only has one loss function, which is different from anytime neural networks that use multiple auxiliary loss functions. Furthermore, the exit policy and the utility function of an agile DNN is different from that of anytime neural networks, and are optimized for resource-constrained systems. Moreover, Zygarde forms an imprecise computing problem where an early exit from the network depends not only on the data but also on the time and energy budget.

4.3. Semi-Supervised $k$-Means Classifiers Construction

In Zygarde, we maintain a semi-supervised $k$-means classifier at each unit. In this section, we describe how we construct and update these classifiers.

Computing Cluster Centroids. Using the trained agile DNN, we obtain a feature representation from each layer for each data point in the training dataset. Using these representations, we train a semi-supervised k-means classifier corresponding to each layer of the agile DNN (see Figure 8). Using the labeled training data, we select the top N features using SelectKBest (George, 2017) and $\chi^{2}$ tests, so that the features are computable on the resource-constrained target device. We utilize the labeled training data to determine the value of $k$ (for k-means) and assign a class label to each cluster. Finally, we compute the centroid of each cluster in the selected feature space.

Determining Utility Threshold. Utility thresholds are crucial to determining whether a data point should exit from the current hidden layer or continue processing through the network. A smaller threshold is likely to force too early exits and thereby, a lower classification accuracy; whereas a larger threshold is like to delay exits and thereby, increase the inference latency. This trade off is demonstrated by Figure 10 for the first layer of the DNN on the CIFAR-100 dataset. For different layers, we see similar trade offs. To determine a suitable utility threshold for each layer, we generate such a trade off curve and pick a utility threshold that ensures a desired minimum inference accuracy as configured by the programmer.

Updating Centroids at Run-Time. We incrementally update the means, i.e., the cluster centroids, of the k-means classifiers at runtime to evolve the classifiers over time and to learn from new examples—which is common in semi-supervised learning approaches (Dhillon et al., 2002; Cheng et al., 2008; Basu et al., 2002). Referring to Figure 8, this is done inside the Classifier Adapter when the classification result from the $k$-means Classifier passes the Utility Test at a unit. A new cluster centroid is computed by taking the weighted average of the current cluster centroid and the current example. Taking the weighted average guards against abrupt changes to the centroids due to the presence of an outlier or incorrect classifications. If the distribution of the input data points changes (e.g., the system is deployed in a new environment), the cluster centroid gradually shifts towards the new mean of the data points as it encounters the new data points.

Updating Centroids beyond Mandatory Layers. Due to early exit from the network, a data sample fails to update the k-means models of the deeper layers. To achieve this, Zygarde adapts the cluster centroids of the deeper layers using the corresponding cluster heads of the layer from which the example exits early. Mathematically, the update operation is c^{i+1}=\frac{1}{r}\sigma\big{(}W^{i+1}\times r\times c^{i}\big{)}. Here, $c^{i}$ and $c^{i+1}$ denote the corresponding cluster centroid of the k-means classifiers of layers $i$ and $i+1$; $W^{i+1}$ denotes weights (including the bias term) for layer $i+1$; $r$ denotes the size of a cluster; and $\sigma(x)=\frac{x+|x|}{2}$ is the non-linear activation function (Schmidhuber, 2015).

Since this technique estimates the cluster centroids of a deeper layer instead of actually running the data samples through those layers, it saves $\mathcal{O}(r)$ multiplication operations and performs the operation in $\mathcal{O}(1)$; at the maximum approximation error of $(\sum_{k=1}^{r}|W^{i+1}\times X_{k}^{i}|-|W^{i+1}\times\sum_{k=1}^{r}X_{k}^{i}|)/(2r)$.

5. Real-Time Scheduler

This section describes the real-time scheduler in Zygarde. First, we introduce an online scheduling algorithm for dynamically-partitioned, sporadic, imprecise tasks on a persistently-powered system. Then, we extend the algorithm for intermittently-powered systems.

5.1. Scheduler for Persistent Systems

Despite being an optimal online scheduling algorithm for sporadic tasks, the earliest deadline first (EDF) algorithm (Krishna, 2001) is not directly applicable to Zygarde as EDF does not consider the accuracy of a DNN. Furthermore, traditional scheduling algorithms for imprecise tasks (Liu et al., 1991; Shi et al., 2009) are not directly applicable to Zygarde as well since the mandatory and optional portions of an agile DNN is determined dynamically at runtime.

To address these challenges, we propose a priority function to prioritize the units of Zygarde. At the end of the execution of an unit, the scheduler selects the highest priority unit as the next unit for execution. The priority function considers not only the remaining deadline of a job, but also the utility (as defined in Section 4.1) and the dynamically determined impreciseness status (i.e., mandatory vs. optional) of a unit:

[TABLE]

where the first term represents the remaining deadline, which is the difference between a job’s absolute deadline $d_{i,j}$ and the current time $t_{c}$. The second term ensures that units with lower utility score $\Psi_{i,j}^{l}$ gets higher priority as these tasks need further execution for accurate classification. The third term is a binary variable $\gamma_{i,j}^{l}\in\{0,1\}$ that denotes if the unit under consideration is mandatory $\gamma_{i,j}^{l}=1$ or optional $\gamma_{i,j}^{l}=0$, which is determined at runtime based on the unit-specific utility threshold. $\alpha$ and $\beta$ are scaling parameters that normalize the deadline and utility, which are the inverse of the maximum deadline and utility, respectively.

Note that, Zygarde supports multiple tasks including multiple DNN tasks as long as the required memory does not exceed the available memory of the system. For non-DNN tasks or other absolute (non-imprecise) tasks, $\gamma_{i}$ is always 1 and $\Psi_{i}$ is a constant for all units. $\Psi_{i}$ is user-defined based on the priority of the task.

5.2. Scheduling for Intermittent System

For an intermittently-powered system, we utilize the $\eta$-factor introduced in Section 3 to extend $\zeta$ as follows:

[TABLE]

Here, $E_{curr}$ is the current energy of the system and $E_{opt}$ is a threshold that determines if the system has enough energy to execute both mandatory and optional units. The expression $\eta E_{curr}$ is high enough to cross the threshold as long as at least one of the two variables $\eta$ and $E_{curr}$ is high-valued and the other is not extremely low. We identify two cases:

First, when $\eta E_{curr}$ is above the threshold, both mandatory and optional units are considered for scheduling. Intuitively, it captures the cases when (a) an energy harvester is predictable and generating at least sufficient energy to keep the capacitor charged, and (b) when an energy harvester is predictable with medium confidence and generating more than sufficient energy. We omit the explanation of the three terms in this case since they are similar to the persistent power system as described in the previous section.

Second, when $\eta E_{curr}$ is below the threshold, only the mandatory units are considered for scheduling. It captures the cases when an energy harvester is – (a) unpredictable, (b) predictable but generates insufficient energy, and (c) predictable with medium confidence and generates sufficient energy.

$\zeta_{I}$ minimizes two types of energy waste in batteryless systems: 1) wasted energy due to executing unnecessary portions of a job, and 2) wasted energy due to not executing any job while the harvester gets enough energy from the source to keep the capacitor charged (Buettner et al., 2011). The first type of waste is avoided by scheduling conservatively when $\eta E_{curr}<E_{opt}$, and the second type of waste is avoided by executing optional units when $\eta E_{curr}\geq E_{opt}$.

5.3. Schedulability Condition

A set of $N$ sporadic tasks is schedulable by an imprecise scheduler if the total utilization, $\sum_{i=1}^{N}\frac{C_{i}}{T_{i}}\leq 1$ (Liu et al., 1991), where the execution time, $C_{i}$ includes only the mandatory portion of the task. Scheduling sporadic jobs in an intermittently-powered system adds further complexity as power outages essentially blocks the CPU and thus increases the CPU utilization by increasing the execution time $C_{i}$, although no task is actually executing on the system as power is out. In order to incorporate energy intermittence into the schedulability analysis framework, we model power outages event as a very high-priority job of a sporadic Energy Task.

In this extended task set having $N+1$ tasks, the schedulability condition becomes $\sum_{i=1}^{N}\frac{C_{i}}{T_{i}}+\frac{C_{e}}{T_{e}}\leq 1$, where, $C_{e}$ and $T_{e}$ are the duration and interval of energy intermittence. The execution time of an energy task, $C_{e}$ is related to the $\eta$-factor of the system. The probability that an energy harvester will remain in its current power-outage state for the next $d$ energy events can be derived from $\eta$ using the properties of a geometric distribution $\eta^{d}(1-\eta)$, whose expected value is $E[C_{e}]=\eta/(1-\eta)$. Given this, the necessary condition for an intermittent computing system to be able to schedule $N$ sporadic tasks is $T_{E}\geq\frac{\eta/(1-\eta)}{1-\sum_{i=1}^{N}(C_{i}/T_{i})}$.

6. Zygarde Programming Model

Zygarde’s programming model consists of: (i) a Network Trainer tool that is used by the developer to train and compress agile DNNs and to generate the $k$-means classifiers and corresponding hyper-parameters; and (ii) APIs for the target embedded device which are used by the developer to write custom C application for an intermittently-powered MSP430 MCU. A high-end development machine is recommended for these one-time, offline steps. At the end of these steps, we obtain an executable binary file for the MSP430 MCU.

6.1. Zygarde Network Trainer

The network trainer takes four inputs from the developer, i.e., (1) a labeled training dataset, (2) DNN architecture/model, (3) timing parameters, and (4) the $\eta$-factor. The network trainer generates C header files as the output – which are used by the APIs for the target embedded platform described in the next section. Figure 11 shows the intermediate steps inside the Zygarde network trainer.

At first, the network trainer trains the agile DNN model using the labeled dataset using the layer-aware loss function described in Section 4.2. It relies on an exhaustive search for hyper-parameter tuning, and outputs the weights and bias parameters of the network. Considering the limited memory of the target device, the DNN is compressed and pruned to reduce its memory requirement (Bhattacharya and Lane, 2016; Chollet, 2017; De Lathauwer et al., 2000a, b; Tucker, 1966; Han et al., 2015; Nabhan and Zomaya, 1994). The network trainer also checks if the compressed network fits into the memory of the target device and signals an error if it does not. Using this compressed agile DNN model and the input dataset, the network trainer generates the cluster centroids and the utility threshold for each layer of the network – following the steps described in Section 4.3.

Finally, C header files are generated that contain the compressed DNN parameters, cluster centroids, utility thresholds, features used in clustering, and task-specific timing parameters (e.g., deadline and period) and the energy parameter (i.e., $\eta$-Factor).

6.2. Zygarde APIs

The Zygarde APIs extend open source SONIC (Gobieski et al., 2019) APIs for intermittent DNN computing by incorporating Zygarde-specific capabilities, such as early termination, cluster-based inference, and scheduling. We divide the APIs into two categories: (1) external APIs, and (2) internal APIs.

The external APIs contain library functions that a developer uses to implement early-exit capable agile DNNs for feature representation and the $k$-means classifiers. These library functions rely on the header files generated by the network trainer to access the classifier parameters and are sufficient for most developers who only want to define the high-level logic of their application. For instance, to implement the two tasks shown in Figure 13 on an MSP430 platform, a developer essentially has to write a C program that uses Zygarde external APIs to implement a state diagram similar to the one shown in Figure 13.

The internal APIs provide some of the lower level functions that are primarily used by the external APIs. These APIs implement several key features of Zygarde including the scheduler, job queue management, time management, and handling the timers. If a developer wishes to change the default implementation of any of these functions, they need to override these methods to provide their own implementation.

7. Implementation

Computing Device. We use TI-MSP430FR5994 (msp, 2018) MCU (shown in Figure 14) that has 256KB of FRAM, 8KB of SRAM, 6-channel DMA, a low energy accelerator (LEA), and an operating voltage range of 1.8V to 3.6V. During the training phase, we use an Intel Core i7 PC with RTX2080 GPU to train and compress the agile DNN, initialize the centroids of semi-supervised $k$-means classifiers, and compute the utility thresholds.

Energy Harvester. Figure 14 shows our solar and RF energy harvester setup. The solar harvester includes an Ethylene Tetrafluoroethylene (ETFE) based solar panel (sol, 2020) and a step-up regulator (ltc, 2015). We use the Powercast harvester-transmitter pair (pow, 2016a, b) to harvest RF energy. Like previous works on intermittent systems (Gobieski et al., 2019; Maeng et al., 2017), both harvesters use a 50mF capacitor.

Sensor Peripheral. We use an electret microphone (BOB, 2018) and the built-in ADC in MSP430 for acoustic sensing. For visual sensing, we use an OV2640 CMOS camera module (cam, 2015) connected via I2C and SPI. Using LEA and DMA, we perform FFT on audio data and write the audio data to the FRAM without involving the CPU.

Time Keeping. Like  (Yıldırım et al., 2018; Hester et al., 2017), we use a real-time clock, DS3231 (tim, 2015) for timekeeping in most of the experiments. We use this clock only during the power up to sync and maintain the internal clocks of the MCU. This clock is easily replaceable with an SRAM or capacitor-based timekeeping system during power outages (Rahmati et al., 2012; Hester et al., 2016). In order to quantify the effect of such a batteryless timekeeper on Zygarde, we implement and use an open-source remanence clock, called CHRT (de Winkel et al., 2020), in one of the experiments. To use the CHRT correctly with Zygarde, the energy required to charge the CHRT has been considered when defining the energy events to estimate the $\eta$-factor.

Libraries. Zygarde uses an open-source intermittent execution model SONIC (Gobieski et al., 2019) and related APIs (e.g., ALPACA (Maeng et al., 2017)). We use Tensorflow (Abadi et al., 2016) for training the DNN models.

8. Microbenchmarks

In this section, we evaluate each component of Zygarde using datasets and compare Zygarde with baseline algorithms. We also observe the effect of capacitor size and remanence clock on Zygarde.

8.1. Datasets and Environments

Datasets and DNNs. To evaluate the performance of different components of Zygarde, we use four datasets: MNIST (LeCun et al., 1998), ESC-10 (Piczak, [n.d.]), CIFAR-100 (Krizhevsky et al., 2009), and Visual Wake Word (Chowdhery et al., 2019). MNIST is a popular image dataset having 80,000 $28\times 28$ pixel images (60,000 for training and 10,000 for testing) and ten classes, and it has been used for evaluating state-of-the-art intermittent computing systems (Gobieski et al., 2019). ESC-10 also has ten classes and 44.1 kHz five seconds-long audio clips. We use 1s audio downsampled to 8KHz. We split the dataset into 80% training and 20% testing datasets. CIFAR-100 contains $32\times 32$ pixel color images from 100 classes. It has 500 training images and 100 testing images per class. In order to fit this dataset in the MSP430, we use randomized subsets of 5 classes from the dataset for 100 iterations and report the average.

Visual Wake Word (VWW) is a large dataset containing 82,783 training and 40,504 validation images from the state-of-the-art vision dataset COCO (Lin et al., 2014). To fit these images into the MCU’s memory, we first crop an image to move the target object (human) in the center and then downsample the cropped image to $32\times 32$ pixels. Our dataset can be accessed at (vww, 2020). Note that if we only downsample the image to $32\times 32$ pixels without cropping it first, the resultant image scales down the target object (human) so much that they are not recognizable anymore. Figure 15 shows an example image from the VWW dataset, followed by two downsampled versions of it – with and without cropping.

We implement four compressed networks summarized in Table 17. Our feature-maps after each layer consist of a maximum of 150 features selected using $k$-best select. These feature-maps are used for the semi-supervised $k$-means classifiers. The scheduler has a queue-size of 3.

Controlled Energy Sources. To evaluate the system with different $\eta$-factors ($\Delta T$=1s and $\Delta K$=9.36mJ), we perform controlled experiments. To determine the value of $\Delta K$, we run the system for multiple iterations and take the highest observed energy consumption. We vary the distance between the transmitter and the receiver between 1-5 feet for RF. We simulate solar power with three dimmable bulbs with varying intensity (5.6 Klx - 35 Klx) as shown in Figure 14. The seven scenarios considered for the evaluation is described in Table 17. Note that, we use outdoor scenarios and windowed rooms to get the sunlight for the real-life experiments in Section 9.

8.2. System Overhead

Figure 18 shows the overhead of different components of Zygarde described in Section 2. To measure the execution time and energy consumption, we use the TI eZ-FET debug probe with EnergyTrace++ (Ene, 2014), which provides milliseconds and $\mu$J resolution data. We isolate each component of Zygarde and report the average overhead from five repeated measurements. To measure smaller overheads we repeat the experiment for multiple iterations (e.g., 2000 iterations for energy manager) and report the mean overhead of a single execution. We use a persistent power source during overhead measurements.

The job generator reads 1s audio data from the microphone, performs FFT, and writes it to the FRAM in 1.325s. The first convolution layer (ESC-10 network of Table 1) has 2.6$\times$-3.6$\times$ higher execution time than other convolution layers due to larger input dimension. Using max-pool with stride decreases the input size, inference time, and energy consumption at each layer. The last fully-connected layer performs 50% less multiplications than the previous layer and thus has a lower cost. Each job executes the semi-supervised $k$-means classifiers at most four times. It is 14$\times$ faster and 13$\times$ more energy-efficient than executing the whole DNN. Execution of the $k$-means classifier includes performing the utility test, classifying with k-means classifier and updating the model centroids for adaptation. For N examples in the system, the scheduler kicks in $4N$ times and the overhead of this specific example with three jobs are 3.72 ms and 636$\mu$J, which is less than 1% of the overall cost of processing an example. The energy manager has negligible cost and runs once every time the scheduler executes.

8.3. Effect of Layer-Aware Loss Function

In Figure 20, we compare our proposed layer-aware loss function with cross-entropy loss (Zhang and Sabuncu, 2018) and contrastive loss (Islam and Nirjon, 2019) functions when early termination is in action. Since loss functions are equally applicable to both persistently-powered and energy-harvested systems, we conduct this experiment in a persistently-powered setting. We train three agile DNNs with three different loss functions that have the same network structure, hyper-parameters, and training dataset. All three networks use the proposed utility test where the utility-threshold is determined during training.

Though the loss functions achieve similar accuracy ($\approx$98% for MNIST and $\approx$75% for ESC-10) without early termination, their performance varies when early termination is applied. Note that, the inference accuracy of ESC-10 suffers due to downsampling of the 5s and 44KHz data samples to 1s and 8KHz data samples. In Figure 20, the layer-aware loss function demonstrates 4.13%-13.40% higher accuracy than cross-entropy loss by forcing the layers to learn distinguishable features (Xie and Tu, 2015). It also decreases the average inference time by upto 13.97%, by executing the final layer of 14%-26% less jobs compared to cross-entropy loss. Layer-aware loss function further achieves 2%-5% higher accuracy and 2%-9% less average inference time than the contrastive loss function. Thus, layer-aware loss function achieves higher accuracy and lower inference time than other loss functions when early termination is active.

8.4. Effect of Early Termination

In Figure 20, we evaluate the proposed utility test by comparing it with a system that does not implement early exit and an oracle that knows the exact number of units needed for each data sample. We use the same persistently-powered system and dataset as in Section 8.3. All of these systems use the same trained network with the layer-aware loss function. Utility-based termination (exit) achieves similar accuracy while lowering the average inference time by 4%-26%. The difference in accuracy between these systems is below 2.5%.

8.5. Performance of the Real-Time Scheduler

We evaluate the proposed scheduling algorithm for dynamic imprecise tasks in both persistently and intermittently powered systems for four different $\eta$-factors and two different CPU utilization described in Table 17. To compare our proposed algorithm, we choose earliest deadline first (EDF) and one of its variants– earliest deadline first mandatory (EDF-M). EDF-M schedules only the mandatory portions of the jobs. We choose EDF as a baseline because it is the optimal online scheduling algorithm for sporadic tasks. Here, both Zygarde and EDF-M use the proposed utility test to partition jobs into mandatory and optional units. For the fairness in comparison, successful completion of a job’s mandatory units before deadline makes the job schedulable in all algorithms. Note that, we discard a job after its deadline to avoid domino effect (Sharma et al., 2014).

Persistently Powered System. Figure 24 shows the performance of proposed scheduling algorithm for MNIST dataset for $T=3s$ and $D=6s$. As the CPU utilization ($U$) is greater than one, none of the schedulers can schedule all the tasks even on persistent power. However, with early termination, EDF-M and Zygarde schedule 17% more jobs. In Figure 24, we schedule 80 jobs from the ESC-10 dataset, where $U<1$, $T=0.36$ minutes, and $D=0.72$ minutes. Persistently powered system (System 1) can schedule all the tasks with EDF, EDF-M, and Zygarde. In Figures 24 and  24, we schedule 500 and 40,000 jobs for CIFAR-100 and visual wake words (VWW) datasets, respectively, where the deadline is twice the period. In both cases, EDF-M and Zygarde schedules all the jobs while EDF fails to do so. As successfully scheduling only the mandatory units of a job before deadline is sufficient to be schedulable, EDF-M schedules similar number of jobs as Zygarde. However, Zygarde achieves higher accuracy by opportunistically executing optional units.

Intermittently Powered Systems. For intermittent systems (Systems 2-7), EDF-M schedules 14.98%-19.51% more jobs for MNIST, 9.44%–20.70% more jobs for ESC, 8.59%–33.59% more jobs for CIFAR, and 16.97%–24.53% more jobs for VWW than EDF. If the utility tests were optimal, EDF-M would have produced correct results for all the scheduled jobs. However, due to the limitation of utility tests, Zygarde increases the number of scheduled jobs that produce the correct results by up to 27.60% by executing some of the optional units. We observe that, Zygarde increases the performance (i.e., the number of scheduled jobs that produce correct results) from EDF-M when $\eta$ is high. With low $\eta$, the performance of Zygarde and EDF-M becomes similar as no optional units are executed. It is interesting to notice that despite having the same $\eta$, solar powered systems schedule 9% - 31% more jobs than RF powered systems due to more available power.

8.6. Effect of Capacitor Size

The goal of this experiment is to quantify the effect of the capacitor’s size on scheduling. We use the CIFAR-100 dataset and its corresponding DNN (Table 17), and power the system from an intermittent RF energy source ($\eta$ = 0.51) at around 0.5m distance. The period of the tasks are varied between 9s to 11s and the deadline is set to twice the period. We use four different capacitors: 0.1mF, 1mF, 50mF, and 470mF. This setup and workload stress tests the system and forces the scheduler to miss the deadline when the capacitor values are too small or too high. Figure 26 shows that when the capacitor value is below 50mF, more tasks miss their deadlines as they re-execute an atomic fragment when the power goes off before its completion. On the other hand, when the capacitor value is high (e.g., 470mF), tasks miss deadline due to the extra time required to charge such a large capacitor. Hence, we choose to use a 50mF capacitor for the rest of the experiments. Note that although we empirically determine a suitable capacitor for our experiments, one can roughly estimate the optimal value of the system capacitor, $C$ by using a capacitor’s energy equation, when the average input power, $P$, voltage across the capacitor, $V$, and the difference between the deadline and the total execution time of the task, $\delta T$, is known: $C=\sqrt{\frac{2P\delta T}{V^{2}}}$.

8.7. Effect of Remanence Clock

Keeping track of the time is crucial for a real-time scheduler and it is a hard problem, in general, for batteryless systems. To keep track of time reliably across power failures, recently, a batteryless remanence clock, namely the Cascaded Hierarchical Remanence Timekeeper (CHRT) (de Winkel et al., 2020) has been proposed for intermittently-powered systems. The CHRT clock has three modes or tiers. Its tier-1 yields near-perfect time-keeping accuracy, but has a range of only 100ms. On the other hand, the tier-3 offers 1s resolution, 100s range, and reports accurate time 80% of the cases, while reporting +1s error for the rest of the time and rarely shows +2s, -1s or -2s error. We implement this clock following the open source hardware design (see Figure 14) and use it to power Zygarde– to implement a completely batteryless system. We evaluate the effect of batteryless CHRT clock on Zygarde’s scheduler and compare it to the performance of Zygarde when it uses battery-powered RTC.

Table 26 shows the number of tasks meeting deadlines for both types of clocks for the systems 2–4 (see Table 17 for definitions). We do not show results for systems 5–7, which are powered by RF harvesters and require using CHRT tier-1 (which is optimized for RF), since the results are identical for both CHRT and RTC. We observe that the number of missed jobs increases with the number of reboots due to intermittent energy. Upon investigating the cause we find that during the positive error of the CHRT clock, the scheduler either reports the missed deadlines or terminates a job early, as it mistakenly thinks that the deadline has passed, and thus, continuing to execute these tasks is a waste of time. During negative error of the CHRT clock, the scheduler schedules a job despite the fact that it missed the actual deadline and triggers a domino effect that results in more tasks missing their deadlines. However, CHRT shows negative error $<3\%$ time and often it compensates for a positive error. Overall, the loss of schedulable tasks due to the use of a batteryless clock is below $0.1\%$.

9. Real-World Application Evaluation

In the previous section, we compared the performance of Zygarde with different baseline algorithms. In this section, we observe Zygarde in two real-world applications. In the first application, we perform acoustic sensing and show how different scenarios affect the system performance. In the second application, we compare the performance of Zygarde with a state-of-the-art intermittent DNN inference system (Gobieski et al., 2019) for visual sensing tasks in a real-world setting.

9.1. Acoustic Sensing

Experimental Setup. In this section, we evaluate Zygarde in real-world uncontrolled experiments using six audio event detection applications. Due to the presence of background noise and multiple audio events in the data, these applications require DNN-based features for audio event representation and classification. Existing works show that DNN performs significantly better than threshold or classic machine learning-based audio event detectors in real-life noisy environments (Kons et al., 2013; esc, 2018).

Table 1 shows the the application environment, energy source, harvester placement, cause of energy intermittence, target event and other events present in the environment for the six applications. Each applications runs for 10 minutes and the audio sensor samples every two seconds. We play recorded sound, that are not used during training, 10 times from a speaker as the positive example. The relative deadline of the jobs are 3s which is the required execution time for the whole model. The agile DNN, consisting of a convolution layer and two fully connected layers, has an execution time that varies between 1.7s and 3s, depending on early termination. As it is not possible to ensure that each audio event of the target classes falls neatly into one second buckets, we combine the outputs of two consecutive jobs by taking their logical OR.

We use two energy harvesters: solar and RF. The solar energy harvester is affected by outdoor influences such as passing vehicles. We vary the distance between the RF transmitter and the receiver to test the applications under different levels of noise and interference.

Results. In Figures 27(a)-(f) we plot the MCU’s input voltage, the cut-off voltage, the classifier’s output, and deadline misses for the six applications over time. Findings from this experiments are as follows.

The car detector in Figure 27(a) always harvests sufficient energy from the sun and meets the deadline for all jobs. However, it misclassifies twice when both pedestrians (talking) and cars are in the scene due to the limitations of the classifier. The dog monitor in Figure 27(b) experience intermittency due to people blocking the sun. It misses two target events due to the lack of sufficient energy to read the sensor data and misclassifies one event due to the limitation of the classifier. For two audio events, the applications experience deadline misses despite doing accurate classification because of the limitation of the utility test. For similar reasons, the people detector in Figure 27(c) fails to sense two events and misclassifies one.

The baby monitor in Figure 27(d), powered with an RF harvester, does not harvest enough energy to read audio samples during one audio event. It also fails to finish execution of mandatory units within the deadline for one event. Due to the limitation of the utility test, it misclassifies one event and misses deadline of another. The Laundry monitor in Figure 27(e) misclassifies one event and misses the deadline for two. The printer monitor in Figure  27(f) experiences the highest intermittence, misses four deadlines and misclassifies three events.

A number of interesting observations from these experiments are: (1) a shorter power-off period decreases the number of event misses, e.g., the solar powered dog monitor misses more events than the laundry monitor despite having less frequent reboots due to insufficient power supply; (2) a shorter continuous energy results in more deadline misses, as evident in dog monitor and printer monitor applications; (3) deadline and target event misses depend on the harvested energy and the accuracy of the utility test, whereas the classification accuracy relies on the competence of the classifier and the accuracy of the utility test, e.g., the car detector misclassifies due to the limitation of the classifier, whereas the dog monitor misses the deadline of two correctly classified samples due to the inaccuracy of the utility test.

9.2. Visual Sensing

Experimental Setup. We evaluate the performance of Zygarde in a multi-tasking scenario having two visual recognition tasks: traffic sign recognition and shape recognition. Both DNNs have two convolution layers and two fully-connected layers, but the convolution layers of the sign recognizer has 8 and 16 filters, whereas the convolution layers of the shape recognizer has 4 and 8 filters. The shape recognizer’s execution time is about half of sign recognizer’s execution time, and hence, it has a smaller relative deadline. After capturing an image, a sign detection job is created and inserted into the job queue, followed by a shape detection job.

Figure 28(a) shows the setup for this experiment. We use a 2MP OV2640 camera sensor and capture the test images from the GTSB (Stallkamp et al., 2012) dataset displayed on the screen of a laptop. We use 80% of the dataset for training and the remaining 20% for testing. We annotate the dataset to label the shape of the sign. We use a solar energy harvester to power the system and acquire 5V and 3V power lines for the camera and the MSP430, respectively, by using two voltage regulators. The camera module requires 4s to capture an event but works in parallel with the MSP430 which uses DMA.

Results. In Figure 28(b), we compare the performance of Zygarde against SONIC’s (Gobieski et al., 2019), which does not implement early termination and uses either an EDF or a round-robin (RR) scheduler. We observe that due to the high energy demand of the camera, 37% of the events are missed and do not enter any of the three systems. Although SONIC-EDF schedules 55% of the jobs that enter the system, it is partial towards the shape recognition jobs since they have earlier deadlines. By choosing the sign recognition job, which has higher execution time, SONIC-RR does not spare sufficient time to execute shape recognition job. SONIC-RR schedules only 11% jobs that enter the system in total, among which, only 1% are shape recognition jobs due to the shorter relative deadline of the shape recognition task. By performing imprecise computing with early termination, Zygarde assigns different priority to the same job at different units. Thus, Zygarde switches between jobs from different tasks and enables fairness. Zygarde schedules 93% of the jobs that enter the system, where 43% are sign recognition jobs and 50% are shape recognition jobs. Zygarde achieves 61% and 85% classification accuracy for sign and shape recognition, respectively, which is is within 2% of the baselines’ that execute the DNNs end-to-end.

10. Related Work

This section describes the existing literature on intermittent computing, timeliness in batteryless systems, DNN compression and partial execution (Anytime Neural Networks), energy harvesting system model, and real-time scheduling for imprecise and mixed critical task set.

Intermittent Computing. Intermittently powered systems experience frequent power failures that reset the software execution and result in repeated execution of the same code and inconsistency in non-volatile memory. Previous works address the progress and memory consistency using software check-pointing (Ransford et al., 2012; Maeng and Lucia, 2018; Hicks, 2017; Lucia and Ransford, 2015; Van Der Woude and Hicks, 2016; Jayakumar et al., 2014; Mirhoseini et al., 2013; Bhatti and Mottola, 2016), hardware interruption (Balsamo et al., 2015, 2016; Mirhoseini et al., [n.d.]), atomic task-based model (Maeng et al., 2017; Colin and Lucia, 2016, 2018) and, non-volatile processors(NVP) (Ma et al., 2017, 2015). Recently SONIC (Gobieski et al., 2018, 2019) proposes a unique software system for intermittent execution of deep neural inference combining atomic task-based model with loop continuation. Despite being used for intermittent execution by Zygarde, SONIC does not handle time-aware, partial execution of DNN with model adaptation like Zygarde.

Timeliness of Batteryless Systems. Prior works on intermittent computing propose runtime systems to increase the likelihood of task completion by finding optimum voltage (Buettner et al., 2011), adapting execution rate (Sorber et al., 2007; Dudani et al., 2002), and discarding stale data (Hester et al., 2017). However, none of these works consider the accuracy/utility of the target application or deadline-aware execution of tasks. Some works in wireless sensors (Zhu et al., 2012; Moser et al., 2006, 2007) have addressed scheduling, but none of them consider the higher computation load of inference tasks. INK (Yıldırım et al., 2018) proposes a reactive kernel that enables energy-aware dynamic execution of multiple threads. Note that INK does not consider DNN tasks or utilize partial execution of tasks for increasing schedulability. Unlike INK, which schedules kernel threads and only one data sample at a time, Zygarde  schedules multiple data samples present in the job queue. Recent work on real-time in intermittent systems (Islam and Shahriar, 2020) explores this problem for only periodic tasks and unlike Zygarde  does not considers the impreciseness of the taskset. Other works focus on maintaining time through power loss (de Winkel et al., 2020; Rahmati et al., 2012; Hester et al., 2016) by exploiting discharge rate of SRAM and capacitors. Our work is complementary to these works and relies on these techniques for timekeeping.

Compression and Partial execution of DNN. Recent works reduce the cost of DNN inference by pruning and splitting models (Han et al., 2015; Nabhan and Zomaya, 1994; Yao et al., 2017; Yao et al., 2018a, b, [n.d.]), reduction of floating-point and weight precision (Johnson, 2018; De Sa et al., 2017; Han et al., 2016), and factorization of computation (Bhattacharya and Lane, 2016; Chollet, 2017; Nakkiran et al., 2015; Szegedy et al., 2017; Szegedy et al., 2015, 2016; Wang et al., 2017). Though these works are crucial for enabling fast DNN execution, they alone are not sufficient for batteryless systems. Binary networks (Courbariaux et al., 2015; Hubara et al., 2016; Rastegari et al., 2016; Courbariaux and Bengio, 2017) are not suitable for batteryless systems due to the higher number of required parameters (Gobieski et al., 2019).

Recent works propose early exit during DNN inference (Teerapittayanon et al., 2016; Bolukbasi et al., 2017; Figurnov et al., 2017; Leroux et al., 2017). However, they are not sufficient for highly constrained batteryless systems due to the large termination overhead. In most cases, the terminations require execution of a neural layer requiring $45\times$ more execution cycles than performing utility test and classification with Zygarde . Moreover, these approaches lack model adaptation capability which is required for life-long sensing. Some works on anytime neural networks depends on module selection  (Teja Mullapudi et al., 2018; Odena et al., 2017; Huang et al., 2017), dynamic layer pruning during inference (Veit and Belongie, 2018; Huang et al., 2016; Bolukbasi et al., 2017; Harvey et al., 2002; Huang et al., 2017) and depth and width adjusting techniques (Yu and Huang, 2019; Karayev et al., 2012; Karayev et al., 2014; Hu et al., 2019; Kim et al., 2018) for anytime prediction. However, none of these works have considered the the effect of energy intermittence and they are yet to be modeled as imprecise task.

Modeling Energy Harvesting Systems. Previous works on energy harvesting (EH) modeling of a specific energy source have achieved promising results in predicting available energy (Crovetto et al., 2014; Sharma et al., 2018; Jia et al., 2018). Some other works have focused on analytically model the trade-off associated with length of history to maximize forward propagation (San Miguel et al., 2018). However, none of the prior works are generalizable and fails to model energy harvesting systems irrespective of energy source. In Zygarde, we provide a single metric, $\eta$ factor that models the predictability of an energy harvesting system irrespective of the source.

Real-time Scheduling for Imprecise and Mixed Critical Task Set. Previous works on imprecise computing only considered the task models where mandatory-optional partitions are fixed and known a priori (Liu et al., 1991; Shi et al., 2009). Existing works on Quality of Service (QoS) based resource management (Rajkumar et al., 1997; Lee et al., 1999) do not handle the data-dependant dynamic relationship between quality and required unit for each job in  Zygarde. Though,  Zygarde can also be formulated as a mixed-critical system, the changing utility of jobs at runtime and intermittent energy makes it more suitable as a imprecise-computing system. Previous works on mixed-critical systems (Baruah et al., 2011, 2010; Vestal, 2007; Li and Baruah, 2010; Dorin et al., 2010; Buttazzo et al., 1995) propose scheduling schemes for predetermines critical levels, which is a characteristic of a task. On the other hand, utility in  Zygarde varies for each job.

11. Discussion

11.1. Importance of DNNs

For batteryless sensing systems, the inference accuracy dictates the response time and the energy-efficiency of the system (Gobieski et al., 2019). Due to very high energy cost of wireless communication, these systems have to implement a large capacitor – which takes several minutes to charge – in order to send just one data packet. Hence, every false positive wastes significant amount of energy and time. This is why, DNNs are preferred over less accurate traditional classifiers such as Support Vector Machines (SVM), K-Nearest Neighbours (KNN), k-means, and Random Forest. Table 2 shows that DNNs are 1%–15% more accurate than the traditional classifiers.

11.2. Generic Utility Functions

In Zygarde, the utility function provides an estimate of how confident the cluster-based classifier is. For a different type of classifier, although the proposed utility function may not be directly applicable, the general principle behind the utility function remains the same. For some classifiers (Alpaydin, 2020), e.g., support vector machine and K nearest neighbour, the distance of the input data point from the decision boundary or the neighbours can be used to design the utility function that is similar to Zygarde’s. For classifiers that provides a probability distribution over all classes as the output (Alpaydin, 2020), e.g., neural networks, naïve bayes, and logistic regression, we recommend using the entropy (Cheng et al., 1999) of this distribution as the utility function, i.e., $U=-\sum_{i=1}^{c}p_{i}{\log}_{2}p_{i}$, where $p_{i}$ is the probability of the input being in class $i$ and $c$ is the total number of classes. A higher entropy indicates that the probability of the input belonging to some class is higher than the rest of the classes, whereas a lower entropy indicates similar probability across all classes.

11.3. Adapting the $k$-Means Classifiers

In Zygarde, we adapt the cluster-based classifiers at runtime since a classifier running on a perpetually-powered system is likely to encounter shifts in the input data distribution over its extended lifetime. To enable this, we implement a simple strategy where the cluster centroids are updated by taking the weighted average of the current centroid and the new data point. By assigning more weights to the current centroid, we ensure that the adaptation process is gradual, and is not affected by a few outliers. This strategy has both benefits and limitations.

The benefit of cluster adaptation is that if a system is trained and tested in different environments, unless measures are taken to adapt the classifier, its accuracy drops. We conduct an experiment to quantify this. We first divide the ESC-10 audio dataset in to 80% training and 20% testing subsets. Then we record only the testing subset in three different environment, i.e., lab, hall, and office. The training subset, 80% data, is recorded only in environment 1 and is used to train the agile DNN and the initial k-means classifier. We test the accuracy of the classifier on the testing subset from environment 1 (lab), followed by testing the accuracy on the testing subset from environment 2 (hall), followed by testing the accuracy on the testing subset from environment 3 (office). We repeat this experiment with and without the cluster adaptation step of Zygarde. Figure 30 shows the result. Without the adaptation, Zygarde loses 8% accuracy due to the environment changes. More than half of this lost accuracy is gained back when Zygarde enabled cluster adaptation.

Two major limitations of this approach are: (1) the adaptation process being slow, if the environment changes rapidly, the system may not be able to adapt fast enough. By adjusting the weights assigned to the new data, this problem can be addressed; (2) the proposed adaptation process is robust to only a certain types of distribution shifts, e.g., translation and rotation of feature spaces, where the relative distances of the cluster heads do not change significantly. However, if the shift in the data distribution in the new environment is complex and/or non-linear, this simple threshold-based cluster adaptation approach may not work. To deal with this, a more sophisticated approach that normalizes the effect of domain shifts in data (Hoffman et al., 2017; Mathur et al., 2019) has to be employed by adding an extra layer of computation prior to the clustering step.

11.4. Limitation of $\eta$ Factor

In Zygarde, we estimate the $\eta$-factor offline, using energy harvesting traces of the systems that we experiment with. The modeling accuracy of $\eta$ largely depends on the length of this empirical study – which must be long enough to capture the variability in harvested energy. Note that this variability depends not only on the energy source, but also on how the system is used by a user or how/where it is deployed. Hence, prior knowledge about the system’s energy usage pattern and/or the experience of the system designer are crucial to determining a reasonable study duration to obtain an accurate estimate of the $\eta$-factor.

A limitation of the offline estimator for $\eta$ in Zygarde is that there is always a possibility that the estimate is off when the system is deployed in the wild. To deal with this scenario, the accuracy of $\eta$ could be assessed at runtime, and then $\eta$ could be updated via an online or an offline re-estimation process. Such an assessment is possible since $\eta$ is used by the system to predict the energy state of the next time slot, for which, the system observes the ground truth immediately after the current time slot, and thus, the system can precisely compute the prediction error at runtime. Depending on the amount of this error, $\eta$ can be adapted to $\eta\pm\delta\eta$, where $\delta\eta$ is proportional to the prediction error.

In our experiments, however, we do not have to perform such adaptations as the empirically derived values for $\eta$ have been fairly accurate. Here, the accuracy of estimation refers to how closely we are able to characterize the randomness in intermittent energy, as opposed to accurately predicting the harvested energy. In Figure 30, we show that the estimated value of $\eta$ for three harvesters (used in Section 3) converges to their respective prediction accuracy values. For example, the kinetic energy harvester’s estimated $\eta$-factor is 0.65, and its (measured) accuracy of predicting the energy state of the next slot is also close to 65%. The convergence of these two values indicates that the estimate is fairly accurate.

11.5. Limitations of the Zygarde Scheduler

Although the scheduler in Zygarde outperforms state-of-the-art scheduling techniques, it has some limitations that need further investigations. First, the scheduler does not provide any guarantee that all the jobs will finish their mandatory part before the deadline. This is primarily due to the uncertainty of the intermittent energy which does not allow us to formally approach the scheduling problem without introducing any probabilistic terms. Besides, in this paper, we only provide a necessary condition for schedulability analysis (Section 5.3), while deriving a sufficient condition remains an open problem. Second, the current design of Zygarde does not schedule the wake-up cycles of the system. Instead, it reactively wakes up (and shuts down) based on the harvested energy/input voltage. Because of this, the system often misses capturing the events as it might be in power down state. By learning the event pattern and incorporating the probability of job arrivals into the scheduling framework, this problem can be addressed. We leave it as one of our future works. Third, the queue size has a significant effect on the scheduler. Due to memory limitations, we cannot implement a longer queue (in Section 8, the queue size is 3). If the queue size is smaller (e.g., 1), the scheduler will only schedules the mandatory portions.

12. Conclusion

In this paper, we introduce a deadline-aware DNN runtime framework for intermittent systems. First, We devise a generic metric, the $\eta$ factor, that models the predictability of an energy harvesting system. Second, we propose a DNN construction and execution technique which adapts the DNN inference process at runtime, and decreases the execution time by 5%-26%. Finally, we utilize the $\eta$ factor and the adaptive execution framework of a DNN to devise an online scheduling algorithm for batteryless systems that successfully schedules 9%-34% more tasks than traditional scheduling algorithms. We also derive a necessary condition for scheduling real-time imprecise DNN tasks on intermittently-powered systems.

Acknowledgments

This paper was supported, in part, by NSF grants CNS-1816213 and CNS-1704469.