Size Matters

TL;DR

This paper investigates how the size of input images affects the performance of fully convolutional neural networks, revealing that each network has a preferred input size linked to the size of features it uses for classification.

Contribution

It uncovers the relationship between input size and model performance, showing that networks are sensitive to input scale and have characteristic feature sizes influencing their accuracy.

Findings

Networks have a preferred input size for optimal performance

Feature size critically influences how inference is distributed across layers

Performance varies non-monotonically with input size

Abstract

Fully convolutional neural networks can process input of arbitrary size by applying a combination of downsampling and pooling. However, we find that fully convolutional image classifiers are not agnostic to the input size but rather show significant differences in performance: presenting the same image at different scales can result in different outcomes. A closer look reveals that there is no simple relationship between input size and model performance (no `bigger is better'), but that each each network has a preferred input size, for which it shows best results. We investigate this phenomenon by applying different methods, including spectral analysis of layer activations and probe classifiers, showing that there are characteristic features depending on the network architecture. From this we find that the size of discriminatory features is critically influencing how the inference process is distributed among the layers.