Solving the Same-Different Task with Convolutional Neural Networks

TL;DR

This paper evaluates convolutional neural networks on the challenging same-different reasoning task, revealing limited benefits of residual and recurrent connections, and achieving super-human performance on several problems.

Contribution

It systematically assesses various CNN architectures on the same-different task, highlighting the limited impact of skip connections and achieving state-of-the-art results.

Findings

Residual and recurrent connections have marginal impact on learning the task.

Older architectures like AlexNet and VGG struggle with the problems.

Newer architectures can generalize better, reaching super-human performance.

Abstract

Deep learning demonstrated major abilities in solving many kinds of different real-world problems in computer vision literature. However, they are still strained by simple reasoning tasks that humans consider easy to solve. In this work, we probe current state-of-the-art convolutional neural networks on a difficult set of tasks known as the same-different problems. All the problems require the same prerequisite to be solved correctly: understanding if two random shapes inside the same image are the same or not. With the experiments carried out in this work, we demonstrate that residual connections, and more generally the skip connections, seem to have only a marginal impact on the learning of the proposed problems. In particular, we experiment with DenseNets, and we examine the contribution of residual and recurrent connections in already tested architectures, ResNet-18, and CorNet-S respectively. Our experiments show that older feed-forward networks, AlexNet and VGG, are almost unable to learn the proposed problems, except in some specific scenarios. We show that recently introduced architectures can converge even in the cases where the important parts of their architecture are removed. We finally carry out some zero-shot generalization tests, and we discover that in these scenarios residual and recurrent connections can have a stronger impact on the overall test accuracy. On four difficult problems from the SVRT dataset, we can reach state-of-the-art results with respect to the previous approaches, obtaining super-human performances on three of the four problems.