Intelligence Beyond the Edge: Inference on Intermittent Embedded Systems

TL;DR

This paper demonstrates a full-scale system for running deep neural network inference on energy-harvesting embedded devices, addressing intermittent power and resource constraints with novel techniques and tools.

Contribution

It introduces SONIC, GENESIS, and TAILS, a suite of methods and tools that enable accurate, energy-efficient DNN inference on intermittently powered, resource-constrained systems.

Findings

SONIC reduces inference energy by 6.9x

TAILS reduces inference energy by 12.2x

System guarantees correct intermittent execution without performance loss

Abstract

Energy-harvesting technology provides a promising platform for future IoT applications. However, since communication is very expensive in these devices, applications will require inference "beyond the edge" to avoid wasting precious energy on pointless communication. We show that application performance is highly sensitive to inference accuracy. Unfortunately, accurate inference requires large amounts of computation and memory, and energy-harvesting systems are severely resource-constrained. Moreover, energy-harvesting systems operate intermittently, suffering frequent power failures that corrupt results and impede forward progress. This paper overcomes these challenges to present the first full-scale demonstration of DNN inference on an energy-harvesting system. We design and implement SONIC, an intermittence-aware software system with specialized support for DNN inference. SONIC introduces loop continuation, a new technique that dramatically reduces the cost of guaranteeing correct intermittent execution for loop-heavy code like DNN inference. To build a complete system, we further present GENESIS, a tool that automatically compresses networks to optimally balance inference accuracy and energy, and TAILS, which exploits SIMD hardware available in some microcontrollers to improve energy efficiency. Both SONIC & TAILS guarantee correct intermittent execution without any hand-tuning or performance loss across different power systems. Across three neural networks on a commercially available microcontroller, SONIC & TAILS reduce inference energy by 6.9x and 12.2x, respectively, over the state-of-the-art.