Position, Padding and Predictions: A Deeper Look at Position Information in CNNs

TL;DR

This paper investigates how position information is encoded in CNNs, revealing that padding influences position encoding and that such encoding can impact the performance on various tasks.

Contribution

It provides the first large-scale analysis of position encoding in CNNs, highlighting the role of padding and border heuristics in semantic representations and task performance.

Findings

Padding drives CNNs to encode position information.

Lack of padding prevents position encoding.

Position encoding can both improve and impair task performance.

Abstract

In contrast to fully connected networks, Convolutional Neural Networks (CNNs) achieve efficiency by learning weights associated with local filters with a finite spatial extent. An implication of this is that a filter may know what it is looking at, but not where it is positioned in the image. In this paper, we first test this hypothesis and reveal that a surprising degree of absolute position information is encoded in commonly used CNNs. We show that zero padding drives CNNs to encode position information in their internal representations, while a lack of padding precludes position encoding. This gives rise to deeper questions about the role of position information in CNNs: (i) What boundary heuristics enable optimal position encoding for downstream tasks?; (ii) Does position encoding affect the learning of semantic representations?; (iii) Does position encoding always improve performance? To provide answers, we perform the largest case study to date on the role that padding and border heuristics play in CNNs. We design novel tasks which allow us to quantify boundary effects as a function of the distance to the border. Numerous semantic objectives reveal the effect of the border on semantic representations. Finally, we demonstrate the implications of these findings on multiple real-world tasks to show that position information can both help or hurt performance.