On the Sustainability of AI Inferences in the Edge

TL;DR

This paper evaluates the performance and energy consumption of various AI models on edge devices like Raspberry Pi and NVIDIA Jetson to inform optimal deployment strategies balancing accuracy, speed, and power use.

Contribution

It provides a comprehensive analysis of model performance and energy trade-offs on multiple edge devices, filling a gap in informed device and model selection for edge AI.

Findings

Neural networks and large language models have distinct resource profiles on edge devices.

Optimization techniques can improve the balance between model accuracy and resource consumption.

Performance and energy metrics vary significantly across device-model combinations.

Abstract

The proliferation of the Internet of Things (IoT) and its cutting-edge AI-enabled applications (e.g., autonomous vehicles and smart industries) combine two paradigms: data-driven systems and their deployment on the edge. Usually, edge devices perform inferences to support latency-critical applications. In addition to the performance of these resource-constrained edge devices, their energy usage is a critical factor in adopting and deploying edge applications. Examples of such devices include Raspberry Pi (RPi), Intel Neural Compute Stick (INCS), NVIDIA Jetson nano (NJn), and Google Coral USB (GCU). Despite their adoption in edge deployment for AI inferences, there is no study on their performance and energy usage for informed decision-making on the device and model selection to meet the demands of applications. This study fills the gap by rigorously characterizing the performance of traditional, neural networks, and large language models on the above-edge devices. Specifically, we analyze trade-offs among model F1 score, inference time, inference power, and memory usage. Hardware and framework optimization, along with external parameter tuning of AI models, can balance between model performance and resource usage to realize practical edge AI deployments.