The Ramifications of Making Deep Neural Networks Compact

TL;DR

Making deep neural networks compact aims to improve energy efficiency, but it can inadvertently increase memory footprint and data movement, potentially reducing overall energy benefits.

Contribution

This paper reveals the indirect effects of compact DNNs on memory usage and energy efficiency, challenging assumptions about their benefits.

Findings

Higher activation-to-parameter ratios increase memory footprint.

Memory footprint can be 15 to 443 times larger than model size.

Compact DNNs may require more bandwidth, reducing energy efficiency.

Abstract

The recent trend in deep neural networks (DNNs) research is to make the networks more compact. The motivation behind designing compact DNNs is to improve energy efficiency since by virtue of having lower memory footprint, compact DNNs have lower number of off-chip accesses which improves energy efficiency. However, we show that making DNNs compact has indirect and subtle implications which are not well-understood. Reducing the number of parameters in DNNs increases the number of activations which, in turn, increases the memory footprint. We evaluate several recently-proposed compact DNNs on Tesla P100 GPU and show that their "activations to parameters ratio" ranges between 1.4 to 32.8. Further, the "memory-footprint to model size ratio" ranges between 15 to 443. This shows that a higher number of activations causes large memory footprint which increases on-chip/off-chip data movements. Furthermore, these parameter-reducing techniques reduce the arithmetic intensity which increases on-chip/off-chip memory bandwidth requirement. Due to these factors, the energy efficiency of compact DNNs may be significantly reduced which is against the original motivation for designing compact DNNs.