Arguments for the Unsuitability of Convolutional Neural Networks for Non--Local Tasks

TL;DR

This paper argues that convolutional neural networks are inherently unsuitable for non-local tasks due to their local processing nature, and suggests attention and iterative methods are necessary for such problems.

Contribution

The paper provides a theoretical argument and lower bounds showing the limitations of CNNs for non-local tasks and advocates for attention mechanisms and iterative processing.

Findings

Convolutional layers are ineffective for global comparison tasks.

A lower bound on parameters needed for non-local tasks is established.

Attention and iterative processing are essential to handle non-local tasks efficiently.

Abstract

Convolutional neural networks have established themselves over the past years as the state of the art method for image classification, and for many datasets, they even surpass humans in categorizing images. Unfortunately, the same architectures perform much worse when they have to compare parts of an image to each other to correctly classify this image. Until now, no well-formed theoretical argument has been presented to explain this deficiency. In this paper, we will argue that convolutional layers are of little use for such problems, since comparison tasks are global by nature, but convolutional layers are local by design. We will use this insight to reformulate a comparison task into a sorting task and use findings on sorting networks to propose a lower bound for the number of parameters a neural network needs to solve comparison tasks in a generalizable way. We will use this lower bound to argue that attention, as well as iterative/recurrent processing, is needed to prevent a combinatorial explosion.