An Empirical Method to Quantify the Peripheral Performance Degradation in Deep Networks

TL;DR

This paper empirically investigates how convolutional padding affects the spatial performance of deep networks, revealing significant degradation near image borders, especially in corners, which impacts real-world applications.

Contribution

It introduces a dataset and methodology to quantify peripheral performance degradation in CNNs, specifically analyzing spatial anisotropy in Mask R-CNN.

Findings

Performance drops near image borders and corners.

Padding does not fully mitigate peripheral degradation.

Spatial anisotropy affects real-world deployment of deep networks.

Abstract

When applying a convolutional kernel to an image, if the output is to remain the same size as the input then some form of padding is required around the image boundary, meaning that for each layer of convolution in a convolutional neural network (CNN), a strip of pixels equal to the half-width of the kernel size is produced with a non-veridical representation. Although most CNN kernels are small to reduce the parameter load of a network, this non-veridical area compounds with each convolutional layer. The tendency toward deeper and deeper networks combined with stride-based down-sampling means that the propagation of this region can end up covering a non-negligable portion of the image. Although this issue with convolutions has been well acknowledged over the years, the impact of this degraded peripheral representation on modern network behavior has not been fully quantified. What are the limits of translation invariance? Does image padding successfully mitigate the issue, or is performance affected as an object moves between the image border and center? Using Mask R-CNN as an experimental model, we design a dataset and methodology to quantify the spatial dependency of network performance. Our dataset is constructed by inserting objects into high resolution backgrounds, thereby allowing us to crop sub-images which place target objects at specific locations relative to the image border. By probing the behaviour of Mask R-CNN across a selection of target locations, we see clear patterns of performance degredation near the image boundary, and in particular in the image corners. Quantifying both the extent and magnitude of this spatial anisotropy in network performance is important for the deployment of deep networks into unconstrained and realistic environments in which the location of objects or regions of interest are not guaranteed to be well localized within a given image.