Smart at what cost? Characterising Mobile Deep Neural Networks in the wild

TL;DR

This paper conducts a comprehensive analysis of mobile deep neural network deployments in real-world Android apps, examining their performance, energy consumption, and deployment practices across diverse devices using a new measurement tool.

Contribution

It presents the first holistic study of DNN usage in the wild on mobile devices, including a new tool, gaugeNN, for automated deployment and analysis across different frameworks and platforms.

Findings

Deep learning models are widely used across popular mobile apps.

There is a significant gap between research techniques and real-world deployment practices.

Mobile DNNs have notable energy footprints and performance variability.

Abstract

With smartphones' omnipresence in people's pockets, Machine Learning (ML) on mobile is gaining traction as devices become more powerful. With applications ranging from visual filters to voice assistants, intelligence on mobile comes in many forms and facets. However, Deep Neural Network (DNN) inference remains a compute intensive workload, with devices struggling to support intelligence at the cost of responsiveness.On the one hand, there is significant research on reducing model runtime requirements and supporting deployment on embedded devices. On the other hand, the strive to maximise the accuracy of a task is supported by deeper and wider neural networks, making mobile deployment of state-of-the-art DNNs a moving target. In this paper, we perform the first holistic study of DNN usage in the wild in an attempt to track deployed models and match how these run on widely deployed devices. To this end, we analyse over 16k of the most popular apps in the Google Play Store to characterise their DNN usage and performance across devices of different capabilities, both across tiers and generations. Simultaneously, we measure the models' energy footprint, as a core cost dimension of any mobile deployment. To streamline the process, we have developed gaugeNN, a tool that automates the deployment, measurement and analysis of DNNs on devices, with support for different frameworks and platforms. Results from our experience study paint the landscape of deep learning deployments on smartphones and indicate their popularity across app developers. Furthermore, our study shows the gap between bespoke techniques and real-world deployments and the need for optimised deployment of deep learning models in a highly dynamic and heterogeneous ecosystem.