Just a Matter of Scale? Reevaluating Scale Equivariance in Convolutional Neural Networks

TL;DR

This paper evaluates how well current convolutional neural networks and new models generalize to different object scales, introduces a benchmark for scale variation, and finds that scale equivariance remains a challenge for robust recognition.

Contribution

It proposes the STIR benchmark for assessing scale generalization and introduces a new model family that enhances scale equivariance in CNNs.

Findings

Models improve scale generalization over standard convolutions

The new models generalize well to larger scales and validate kernel selection consistency

Performance drops significantly with large scale differences, indicating ongoing challenges

Abstract

The widespread success of convolutional neural networks may largely be attributed to their intrinsic property of translation equivariance. However, convolutions are not equivariant to variations in scale and fail to generalize to objects of different sizes. Despite recent advances in this field, it remains unclear how well current methods generalize to unobserved scales on real-world data and to what extent scale equivariance plays a role. To address this, we propose the novel Scaled and Translated Image Recognition (STIR) benchmark based on four different domains. Additionally, we introduce a new family of models that applies many re-scaled kernels with shared weights in parallel and then selects the most appropriate one. Our experimental results on STIR show that both the existing and proposed approaches can improve generalization across scales compared to standard convolutions. We also demonstrate that our family of models is able to generalize well towards larger scales and improve scale equivariance. Moreover, due to their unique design we can validate that kernel selection is consistent with input scale. Even so, none of the evaluated models maintain their performance for large differences in scale, demonstrating that a general understanding of how scale equivariance can improve generalization and robustness is still lacking.