NEUKONFIG: Reducing Edge Service Downtime When Repartitioning DNNs

TL;DR

This paper introduces NEUKONFIG, a framework that significantly reduces edge service downtime during DNN repartitioning by using dynamic switching, enabling near-zero downtime with increased memory usage.

Contribution

The paper proposes a novel dynamic switching approach for DNN repartitioning that minimizes edge service downtime compared to traditional pause-resume methods.

Findings

Reduces downtime from 6 seconds to less than 1 millisecond with increased memory.

Demonstrates effectiveness on a lab-based testbed.

Offers a trade-off between memory usage and downtime.

Abstract

Deep Neural Networks (DNNs) may be partitioned across the edge and the cloud to improve the performance efficiency of inference. DNN partitions are determined based on operational conditions such as network speed. When operational conditions change DNNs will need to be repartitioned to maintain the overall performance. However, repartitioning using existing approaches, such as Pause and Resume, will incur a service downtime on the edge. This paper presents the NEUKONFIG framework that identifies the service downtime incurred when repartitioning DNNs and proposes approaches for reducing edge service downtime. The proposed approaches are based on 'Dynamic Switching' in which, when the network speed changes and given an existing edge-cloud pipeline, a new edge-cloud pipeline is initialised with new DNN partitions. Incoming inference requests are switched to the new pipeline for processing data. Two dynamic switching scenarios are considered: when a second edge-cloud pipeline is always running and when a second pipeline is only initialised when the network speed changes. Experimental studies are carried out on a lab-based testbed to demonstrate that Dynamic Switching reduces the downtime by at least an order of magnitude when compared to a baseline using Pause and Resume that has a downtime of 6 seconds. A trade-off in the edge service downtime and memory required is noted. The Dynamic Switching approach that requires the same amount of memory as the baseline reduces the edge service downtime to 0.6 seconds and to less than 1 millisecond in the best case when twice the amount of memory as the baseline is available.