Not All Ops Are Created Equal!

TL;DR

This paper highlights that traditional metrics like operation count and parameters are insufficient for evaluating neural network efficiency, as actual deployment metrics like energy and memory vary significantly across operation types and must be considered.

Contribution

It demonstrates the discrepancy between common efficiency metrics and real-world deployment metrics, emphasizing the importance of considering energy, memory, and operation types in neural network design.

Findings

Throughput and energy vary up to 5X across operation types.

Activation memory requirements are crucial for network architecture exploration.

Traditional metrics do not fully capture deployment efficiency.

Abstract

Efficient and compact neural network models are essential for enabling the deployment on mobile and embedded devices. In this work, we point out that typical design metrics for gauging the efficiency of neural network architectures -- total number of operations and parameters -- are not sufficient. These metrics may not accurately correlate with the actual deployment metrics such as energy and memory footprint. We show that throughput and energy varies by up to 5X across different neural network operation types on an off-the-shelf Arm Cortex-M7 microcontroller. Furthermore, we show that the memory required for activation data also need to be considered, apart from the model parameters, for network architecture exploration studies.