From Compass and Ruler to Convolution and Nonlinearity: On the Surprising Difficulty of Understanding a Simple CNN Solving a Simple Geometric Estimation Task

TL;DR

This paper investigates the interpretability challenges of a simple CNN trained on a geometric estimation task, revealing insights into the roles of invariance, activation functions, and network weights through theoretical analysis.

Contribution

It provides a detailed theoretical understanding of how a basic CNN learns to solve a simple geometric problem, highlighting the importance of invariance and nonlinearities.

Findings

Understanding trained CNNs on simple tasks is surprisingly difficult.

Theoretical analysis reveals the roles of filters, weights, and nonlinearities.

Invariance and activation shape are fundamental to CNN performance.

Abstract

Neural networks are omnipresent, but remain poorly understood. Their increasing complexity and use in critical systems raises the important challenge to full interpretability. We propose to address a simple well-posed learning problem: estimating the radius of a centred pulse in a one-dimensional signal or of a centred disk in two-dimensional images using a simple convolutional neural network. Surprisingly, understanding what trained networks have learned is difficult and, to some extent, counter-intuitive. However, an in-depth theoretical analysis in the one-dimensional case allows us to comprehend constraints due to the chosen architecture, the role of each filter and of the nonlinear activation function, and every single value taken by the weights of the model. Two fundamental concepts of neural networks arise: the importance of invariance and of the shape of the nonlinear activation functions.