How to Reach Real-Time AI on Consumer Devices? Solutions for Programmable and Custom Architectures

TL;DR

This paper explores cross-stack design techniques to enable real-time AI inference on consumer devices, addressing hardware heterogeneity, computational costs, and accuracy challenges for both programmable processors and custom accelerators.

Contribution

It presents diverse model-, system-, and hardware-level solutions for efficient AI deployment, emphasizing the integration of these techniques for real-time performance on commodity devices.

Findings

AI systems can run efficiently without overburdening mobile hardware

Custom accelerators like ASICs and FPGAs enable next-generation AI applications

Cross-stack solutions improve inference accuracy while maintaining real-time performance

Abstract

The unprecedented performance of deep neural networks (DNNs) has led to large strides in various Artificial Intelligence (AI) inference tasks, such as object and speech recognition. Nevertheless, deploying such AI models across commodity devices faces significant challenges: large computational cost, multiple performance objectives, hardware heterogeneity and a common need for high accuracy, together pose critical problems to the deployment of DNNs across the various embedded and mobile devices in the wild. As such, we have yet to witness the mainstream usage of state-of-the-art deep learning algorithms across consumer devices. In this paper, we provide preliminary answers to this potentially game-changing question by presenting an array of design techniques for efficient AI systems. We start by examining the major roadblocks when targeting both programmable processors and custom accelerators. Then, we present diverse methods for achieving real-time performance following a cross-stack approach. These span model-, system- and hardware-level techniques, and their combination. Our findings provide illustrative examples of AI systems that do not overburden mobile hardware, while also indicating how they can improve inference accuracy. Moreover, we showcase how custom ASIC- and FPGA-based accelerators can be an enabling factor for next-generation AI applications, such as multi-DNN systems. Collectively, these results highlight the critical need for further exploration as to how the various cross-stack solutions can be best combined in order to bring the latest advances in deep learning close to users, in a robust and efficient manner.