Comparison and Benchmarking of AI Models and Frameworks on Mobile Devices

TL;DR

This paper introduces AIoTBench, a comprehensive benchmark suite for evaluating and ranking the inference capabilities of mobile and embedded devices using diverse models and frameworks.

Contribution

The paper presents a new benchmark suite, AIoTBench, with unified metrics for assessing AI inference performance on mobile devices, covering multiple models and frameworks.

Findings

Compared and ranked 5 mobile devices using AIoTBench.

Proposed two unified metrics: VIPS and VOPS.

Benchmark results highlight performance differences across devices.

Abstract

Due to increasing amounts of data and compute resources, deep learning achieves many successes in various domains. The application of deep learning on the mobile and embedded devices is taken more and more attentions, benchmarking and ranking the AI abilities of mobile and embedded devices becomes an urgent problem to be solved. Considering the model diversity and framework diversity, we propose a benchmark suite, AIoTBench, which focuses on the evaluation of the inference abilities of mobile and embedded devices. AIoTBench covers three typical heavy-weight networks: ResNet50, InceptionV3, DenseNet121, as well as three light-weight networks: SqueezeNet, MobileNetV2, MnasNet. Each network is implemented by three frameworks which are designed for mobile and embedded devices: Tensorflow Lite, Caffe2, Pytorch Mobile. To compare and rank the AI capabilities of the devices, we propose two unified metrics as the AI scores: Valid Images Per Second (VIPS) and Valid FLOPs Per Second (VOPS). Currently, we have compared and ranked 5 mobile devices using our benchmark. This list will be extended and updated soon after.