Non-discriminative data or weak model? On the relative importance of data and model resolution

TL;DR

This paper investigates how input and internal resolutions affect neural network performance, revealing internal resolution as the key factor, and introduces Isometric Neural Networks that maintain fixed internal resolution for efficiency and accuracy.

Contribution

The paper demonstrates that internal resolution is more critical than input resolution for neural network performance and introduces Isometric Neural Networks with fixed internal resolution for improved efficiency.

Findings

Internal resolution is the main driver of model quality.

Isometric Neural Networks achieve high accuracy with fewer parameters.

Maintaining fixed internal resolution reduces activation footprint.

Abstract

We explore the question of how the resolution of the input image ("input resolution") affects the performance of a neural network when compared to the resolution of the hidden layers ("internal resolution"). Adjusting these characteristics is frequently used as a hyperparameter providing a trade-off between model performance and accuracy. An intuitive interpretation is that the reduced information content in the low-resolution input causes decay in the accuracy. In this paper, we show that up to a point, the input resolution alone plays little role in the network performance, and it is the internal resolution that is the critical driver of model quality. We then build on these insights to develop novel neural network architectures that we call \emph{Isometric Neural Networks}. These models maintain a fixed internal resolution throughout their entire depth. We demonstrate that they lead to high accuracy models with low activation footprint and parameter count.