Improved Decision Module Selection for Hierarchical Inference in Resource-Constrained Edge Devices

TL;DR

This paper enhances hierarchical inference strategies for resource-limited edge devices, improving decision-making for offloading tasks to balance accuracy and efficiency in tinyML applications.

Contribution

It introduces three novel hierarchical inference approaches and evaluates their effectiveness for image classification on resource-constrained devices.

Findings

Proposed methods outperform traditional local and offloading strategies.

Hierarchical inference improves accuracy and reduces latency.

Evaluation demonstrates benefits in IoT and microcontroller scenarios.

Abstract

The Hierarchical Inference (HI) paradigm employs a tiered processing: the inference from simple data samples are accepted at the end device, while complex data samples are offloaded to the central servers. HI has recently emerged as an effective method for balancing inference accuracy, data processing, transmission throughput, and offloading cost. This approach proves particularly efficient in scenarios involving resource-constrained edge devices, such as IoT sensors and micro controller units (MCUs), tasked with executing tinyML inference. Notably, it outperforms strategies such as local inference execution, inference offloading to edge servers or cloud facilities, and split inference (i.e., inference execution distributed between two endpoints). Building upon the HI paradigm, this work explores different techniques aimed at further optimizing inference task execution. We propose and discuss three distinct HI approaches and evaluate their utility for image classification.