DynaSplit: A Hardware-Software Co-Design Framework for Energy-Aware Inference on Edge

TL;DR

DynaSplit is a dynamic hardware-software co-design framework that optimizes energy efficiency and latency for neural network inference on edge devices through offline optimization and online scheduling.

Contribution

It introduces a two-phase approach combining multi-objective optimization and real-time scheduling to adaptively configure edge computing systems for neural network inference.

Findings

Energy consumption reduced by up to 72%.

Latency maintained at ~90% of user threshold.

Effective adaptation to dynamic workloads.

Abstract

The deployment of ML models on edge devices is challenged by limited computational resources and energy availability. While split computing enables the decomposition of large neural networks (NNs) and allows partial computation on both edge and cloud devices, identifying the most suitable split layer and hardware configurations is a non-trivial task. This process is in fact hindered by the large configuration space, the non-linear dependencies between software and hardware parameters, the heterogeneous hardware and energy characteristics, and the dynamic workload conditions. To overcome this challenge, we propose DynaSplit, a two-phase framework that dynamically configures parameters across both software (i.e., split layer) and hardware (e.g., accelerator usage, CPU frequency). During the Offline Phase, we solve a multi-objective optimization problem with a meta-heuristic approach to discover optimal settings. During the Online Phase, a scheduling algorithm identifies the most suitable settings for an incoming inference request and configures the system accordingly. We evaluate DynaSplit using popular pre-trained NNs on a real-world testbed. Experimental results show a reduction in energy consumption up to 72% compared to cloud-only computation, while meeting ~90% of user request's latency threshold compared to baselines.