Evaluating CNNs on the Gestalt Principle of Closure

TL;DR

This paper evaluates the ability of CNNs like AlexNet and GoogLeNet to recognize Kanizsa triangles, revealing that perceiving closure is challenging for these networks but they can adapt to it.

Contribution

It introduces an evaluation of CNNs on Gestalt principles, specifically closure, using datasets of valid and invalid Kanizsa triangles, highlighting their limitations and adaptability.

Findings

CNNs struggle with classifying Kanizsa triangles based on closure.

Networks show some adaptation to the Gestalt principle of closure.

Perception of closure remains a challenge for current CNN architectures.

Abstract

Deep convolutional neural networks (CNNs) are widely known for their outstanding performance in classification and regression tasks over high-dimensional data. This made them a popular and powerful tool for a large variety of applications in industry and academia. Recent publications show that seemingly easy classifaction tasks (for humans) can be very challenging for state of the art CNNs. An attempt to describe how humans perceive visual elements is given by the Gestalt principles. In this paper we evaluate AlexNet and GoogLeNet regarding their performance on classifying the correctness of the well known Kanizsa triangles, which heavily rely on the Gestalt principle of closure. Therefore we created various datasets containing valid as well as invalid variants of the Kanizsa triangle. Our findings suggest that perceiving objects by utilizing the principle of closure is very challenging for the applied network architectures but they appear to adapt to the effect of closure.

Tables

Table 1. Table 1: Summary of test error rates on various datasets [%]. We trained the CNNs until they converged and evaluated the test sets with a model where the losses on the train and validation set became more stable. The number of epochs that were required to reach this state are indicated in the second column.

	# epochs		OFFS	ANGLE	ROT
AlexNet	80	OFFS	0.7	6.4	10.1
AlexNet	20	ANGLE	48.8	1.2	4.8
AlexNet	8	ROT	50	4.3	0.3
GoogLeNet	204	OFFS	0.7	1.4	14.5
GoogLeNet	8	ANGLE	50	0	0
GoogLeNet	6	ROT	50	1	0.1

Full text

1 Introduction and Related Work

Convolutional neural networks have gained enormous interest in industry and research over the past years because they provide outstanding performance in many visual classification tasks. The basic architecture of a CNN was first introduced by LeCun et al. [2] in 1989. Almost a decade later LeCun et al. [3] created LeNet-5 which was able to classify handwritten digits with an accuracy exceeding 99% on the MNIST dataset. In 2012 Krizhevsky et al. managed to train a deep CNN – later known as AlexNet – to classify 1.2 million images into 1000 different classes with an impressive top-5 test error rate of 15%. In 2014 Szegedy et al. [6] introduced the famous Inception architectures which are also known as GoogLeNet in hommage to LeNet. In this paper we experiment with AlexNet and GoogLeNet111In our case: Inception v3.

Gestalt psychology explains different perceptual phenomena. In 1923 Wertheimer [7] described a set of rules which are essential for our perception of objects, the so-called Gestalt principles. One of these principles is given by the principle of closure, which states that humans tend to fill visual gaps to perceive objects as being whole, even when fragments are missing. Another principle is given by the principle of similarity which states that shapes that are similar to each other tend to be perceived as a unit. Stabinger et al. [4] could show that neither LeNet nor GoogLeNet are capable of comparing shapes. Further experiments were performed in the context of the principle of symmetry, where Stabinger et al. [5] found variations of a dataset which seem to be at the border of what CNNs can do. Kim et al. [1] just recently adapted tools which are used in psychology to study human brains to analyse the neural responses within CNNs to see whether they utilize the principle of closure. They showed that under certain circumstances neural networks do respond to closure effects. In this paper we also experiment with the principle of closure by evaluating the ability of the given CNNs to decide whether an image contains a valid or an invalid Kanizsa triangle. Our goal is to gain more insight from a practical point of view into how challenging it is for a CNN to exploit the closure effect and how well it performs.

2 Evaluating CNNs on Datasets Utilizing the Principle of Closure

2.1 Kanizsa Triangle Dataset

For our tests we generated datasets with 50.000 images each22230.000 training, 10.000 validation and 10.000 test images; dimension: $256\times 256$ px. One half of each dataset shows valid Kanizsa triangles, the other half invalid variants of the Kanizsa triangle. We created the following scenarios (see Fig. 1).

OFFS: Translate one of the vertices by a random offset333To maximize the visual error, the offset is applied in the direction of the connecting line of the other two vertices.. 2. 2.

ANGLE: Change the opening angle within one vertex. 3. 3.

ROT: Rotation of one to three vertices by a random angle. 4. 4.

COMB: Each invalid Kanizsa triangle contains exactly one of the errors out of the set $\{\operatorname{\bf{ROT}},\operatorname{\bf{ANGLE}},\operatorname{\bf{OFFS}}\}$.

2.2 Evaluating the CNNs

It took AlexNet 140 and GoogLeNet 99 epochs444All CNNs were trained using NVIDIA DIGITS https://developer.nvidia.com/digits with the Torch backend and def. settings: fixed learning rate = 0.01, solver = SGD. to correctly classify 95% of the validation set of COMB, which suggests that this problem is rather challenging for the observed CNN architectures. As a reference we used a subset555While the original MNIST dataset contains 60.000 training images and 10.000 validation images, we moved 10.000 training images to a test image set, and deleted 20.000 of the training images while not changing the distribution of the images among the classes. We did this to be comparable with our own datasets. of the MNIST dataset to train AlexNet and GoogLeNet. It took AlexNet only one epoch and GoogLeNet twelve epochs to achieve a test error rate of less than 5%. Please note that one needs to distinguish between ten different classes in the MNIST dataset, while the datasets in this paper only consist of two classes.

To gain more insight, we split the problem into its components and look at the number of epochs the CNNs need to accomplish an accuracy of at least 95% on the validation set666Trained multiple times; lowest results are displayed.:

OFFS: AlexNet 27 epochs, GoogLeNet 188 epochs. 2. 2.

ANGLE: AlexNet 12 epochs, GoogLeNet 6 epochs. 3. 3.

ROT: AlexNet 6 epochs, GoogLeNet 5 epochs.777In further experiments we also worked with a set where only one of the three vertices was rotated. AlexNet needed at least 109 epochs to be able to classify 95% correctly, while GoogLeNet needed 6 epochs.

We observe that GoogLeNet needs many epochs to reach sufficient accuracy on OFFS, and AlexNet requires significantly more epochs to learn to classify OFFS than to correctly classify ANGLE and ROT. Furthermore, we want to point out that AlexNet needed many trials before being able to find a satisfying classifier on the problems involving angles, while GoogLeNet needed many trials on the OFFS dataset.

3 Interpretation and Further Results

Although the problems seem to be very similar at first glance, ANGLE and ROT are locally solvable. Basically it suffices to detect the opening angle and its orientation for each vertex. Then, without considering the position of the vertices, comparing these features leads to the classification result. For OFFS the CNN needs to use higher-level features since it is not able to decide locally anymore. To make this more evident, if we consider the classification matrices in Table 1, we observe that:

CNNs trained on ANGLE are also able to classify ROT and vice versa, but they are not able to classify OFFS above chance. 2. 2.

CNNs trained on OFFS are able to classify ANGLE, as well as ROT above chance although they did not encounter any of these problems before.

4 Conclusion

Our findings suggest that in order to discriminate classes in which the positions of the objects matter, the CNNs need to detect higher-level features and generalize. Apparently, CNNs trained on such a problem set also have significantly lower test error rates on previously unseen perturbations of the data, which suggests that – in our case – they are exploiting the principle of closure. Our experiments show that training on these features is quite challenging, needing significantly more epochs than training on the MNIST dataset. We believe that further analysis of the performance of CNNs in the context of the Gestalt principles is a promising area for future research to gain a better understanding of the differences and similarities between human and artificial neural network perception.